Rescuing Resque again (2015) | Hacker News Reader

Rescuing Resque again (2015)

words.steveklabnik.com

47 pointspolysaturate41 comments

In my opinion, Resque so far failed because like a lot of OSS projects (and unlike Sidekiq), it isn't built on sustainable economical grounds - from the maintainer POV.

Bringing outside "funding" can work for a while (see JRuby), but when the funding stops, it only underlines that the financial engine should really be carefully "designed" at the core of the OSS project, for things to be maintainable on the very long run. Otherwise history is bound to repeat itself.

Large projects can enjoy a large pool of contributors, sometimes with less risks of burn-out (often seen on small OSS projects around me at least), each contributing marginally, with a few paid core members (either directly or indirectly via by-products of a well paid activity), but for smaller projects like Resque, it's more likely a single individual full-time, paid by the project.

Again Sidekiq is really a great inspiration here for me (disclaimer: I plan to launch a "pro" offering for an OSS project I manage at http://kiba-etl.org/, and maintained another open-source for a while before which ultimately went to the cesspit).

Yup, this is exactly it. The only reason I was able to do all the open source work I did was because of my job at Jumpstart Labs, and once that ended, well...

That said, as you mentioned, this is pretty much the default state for the large, large part of open source projects. Something's got to give somewhere, eventually.

What is your take on the Sidekiq model? I'm really curious to hear more about that :-)

I see a number of people around me currently thinking about putting a "fair salary" at the core of their future open-source work. Often it completely changes what the delivered value should ultimately be etc (something I think Mike did think about - I believe).

Do you have thoughts on that approach?

My thoughts are mostly "I'm glad it's working out for Mike, and I hope others can follow his path." I personally have taken pursued the patron model, but that also is because with Rust, that's the only real way to do it.

By patron model, to clarify, you mean a single large company paying most of the bills and bringing funding, right? (either because they want to promote innovation, or this generates financial gains, etc etc)?

I would also think that for a language, this is probably the best model (even if it failed with IronRuby, or if JRuby funding sources moved over time etc).

This is a fairly large endeavour, so it makes sense (at least I believe it!).

Yes. Though I mean me personally, not the language itself. I wanted to work on Rust full time, Mozilla wanted to hire someone to work on Rust full time, it fits.

It's great that it works for you one on one, sincerely! I hope though that we will see more occurrences of OSS projects where N customers are sending money to directly finance the sustainability of the OSS product, just because it delivers value for them (not a moral obligation or PR etc - note that I'm not stating that Mozilla is doing this!).

I feel for libraries & components (unlike languages maybe, as we said), at that specific scale, it can work very well.

I do too.

Hi all! Happy to answer any questions about this effort. I'm targeting 1.26.0 sometime in the next couple of weeks.

Thanks for all your hard work on this over the years!

Something I've wondered for awhile: what are the benefits of continuing Resque development, rather than focusing on making the migration path to Sidekiq as painless as possible? Put another way, what do you hear from users of Resque (like Shopify, as you mentioned in another comment) as the reason that they don't want to migrate off of it?

I was curious about this way back in 2012 when you announced work on Resque 2.0, and I had already made the switch to Sidekiq. I thought there must be a good reason to keep working on it, but I wasn't sure what it was, and I'm still not sure, so I figured I'd just ask.

Sidekiq has a lot of attention amongst new developers, but has at least an order of magnitude less users, at least, if gem downloads are something to go by. So from my standpoint, those people deserve bugfixes, and "throw away what works just to transition to something new" isn't really a great response.

Threading vs process-based jobs is a tradeoff. Threads are not universally better.

As of today, Resque has 6,308,611 downloads. Sidekiq has 5,669,265 downloads. Can you explain why you think "order of magnitude"?

Hey Mike! Sorry, I 1) mis-read that number, 2) when we used to release more often, the download counts for latest versions used to be 10x yours. I don't think that's true at this point. I will update that cache in my brain and not say this again, I apologize.

No worries, I just wanted to make sure you didn't have access to super secret Ruby usage numbers that I didn't!

I wish. :)

Thanks for the quick answer. Makes sense.

Is Github still using Resque?

They are as far as I know, though they've been on a fork for a while, due to https://github.com/resque/resque/issues/552

One employee has mentioned that they're willing to work on it: https://github.com/resque/resque/issues/1372#issuecomment-17...

I have to be honest here. My initial thought when reading this a couple of weeks back was that some things are better left to dead.

Resque has been an example of how not to run an open source project for a while now. (3+ years).

We were working with Resque for a long while and had a lot of issues with performance.

Sidekiq is awesome and it's being well managed and releases are frequent.

But, after thinking about it, I think that Resque has it's place in the community, I definitely think that more than a single project doing this task is needed.

Many people are already familiar with Resque and trust it, those people should be taken care of.

I wish everyone involved a lot of luck with it.

This is another good answer to the question[0] I just posed to Steve. You're right that competition is generally good. On the other hand, it would (arguably?) be much more "Rails-y" to have a single conventional solution that works well and gets a lot of eyeballs. Neither Resque nor Sidekiq are that thing though. It has always struck me as a bit odd that Rails outsources the decision on this component, which nearly every Rails project needs. I think previously it was probably to avoid an external dependency (on Redis), but perhaps that could be reconsidered now that >5.0 has eschewed that avoidance.

Having said all that, one of the things that is most exciting to me about Elixir/Phoenix is that this sort of thing doesn't even come up, because a great and mature solution is built directly into the runtime.

[0]: https://news.ycombinator.com/item?id=10911600

Newer Rails has ActiveJob: https://github.com/rails/rails/tree/master/activejob

Thanks!

  > Resque has been an example of how not to run an open source project
  > for a while now. (3+ years).

Resque is the story of a gem that _everyone_ uses, yet nobody invests in. So the maintainers burn out, and burn out, and burn out. Chris, Terence, Me... trying not to do that again ;)

The problem actually started before you guys came to the Resque :)

I remember you commented on my open source project (Resque plugin), I was impressed with the reach of how deep you want to go with this.

I agree with what you said 100%, everyone is using it and no one invests in it. That's why it's hard to maintain and that's why Sidekiq is "winning". Since money is a justified investment to any company for that project and it keeps the project going.

When i started my project, I used Resque but almost immediately switched to Sidekiq a few months later. I understand it's almost impossible to run into thread safety issues with Resque, but if you're doing anything CPU intensive, Sidekiq really offers much higher performance. The maintainer also makes a living off Sidekiq, selling it to businesses and enterprises too so you can expect more frequent updates.

> but if you're doing anything CPU intensive, Sidekiq really offers much higher performance

The threading model (LWPs vs. forking) should have no impact whatsoever on how quickly a CPU-intensive task completes.

Unless it's a short-lived CPU intensive task, in which case in matters a great deal.

The overhead of fork() is negligible, well under 0.1ms.

There's also the issue, in ruby, that CoW doesn't really work due to the fact that the GC dirties all pages.

This was fixed in Ruby 2.0.

It matters how many of these tasks can be completed in a fixed amount of time. Creating 50 processes requires more resources than 1 process with 50 threads.

Yeah, Sidekiq is an obvious switch once you find yourself running 20 500MB Rails processes just because you want to be able to handle a few simple tasks concurrently. If you have a few big, long-running, CPU-intensive background jobs, then the benefits are less clear, but if you have lots of small, short running jobs where you're mostly interested in throughput, then Sidekiq makes a lot more sense.

Resque-pool (https://github.com/nevans/resque-pool) -- which every Resque user should be using -- loads your process first, then forks the workers. Forking uses copy-on-write semantics, so as long as you're loading everything before forking, you're conserving a great deal of otherwise wasted memory on what would be shareable pages.

The "again" in the title confused me. I knew that I read an article along these lines a while back so I thought this is yet another one. Turns out it was this post, which is dated October 23, 2015.

Aw poo...missed the date when I read it, and haven't seen it before. I'll update the title.

It's again because it's calling back to this one from 2012! http://blog.steveklabnik.com/posts/2012-09-22-resque--let-s-...

I almost alway use Que [1] now. It uses postgres advisory locks instead of Redis so one less dependency to worry about. You probably only need Redis if you're running a tonne of jobs.

[1] https://github.com/chanks/que

If you're already using Resque (the audience for the blog post), you're already using Redis, so Sidekiq would make more sense. Additionally, a quick glance at the syntax for a que job appears that you'd need to change more code (I could be wrong), and the available UI seems to lag Sidekiq's.

Reading this: "Que's job table undergoes a lot of churn when it is under high load, and like any heavily-written table, is susceptible to bloat and slowness if Postgres isn't able to clean it up. The most common cause of this is long-running transactions, so it's recommended to try to keep all transactions against the database housing Que's job table as short as possible." I wonder what consistutes "high load". I throw hundreds of thousands of jobs atop Sidekiq often; if you're using the same database to manage your jobs, it seems that would dogpile your database (since each of those jobs is likely making db calls, and if it's a Rails app, you're typically using the same database)

Rails 5 will require Redis so in the future it will be a dependency anyway ...

Rails 5 does not require Redis unless you want to use Action Cable.

Interesting to note that the current stable branch has commits a couple of days old.

Yes, I've been doing some work, and so have some other contributors. Shopify is upstreaming some patches so they can get back on master.

Reply on news.ycombinator.com