Show HN: Dramatiq – An alternative to Celery | Hacker News Reader

Show HN: Dramatiq – An alternative to Celery

dramatiq.io

53 pointsBogdanp31 comments

> Dramatiq is licensed under the AGPL and it officially supports Python 3.6 and later. Commercial Licensing is also available.

Oh.

> I’ve done a ton of open source work over the course of my career. Companies have used that work to generate income for themselves. Inevitably, I’ve ended up supporting that software on those companies’ behalf for free and that is not sustainable long term.

This is totally 100% true and understandable. Unfortunately, a lot of devs are going to go "hmm, I could battle to convince someone that can approve purchases that it's worth forking out $2k/year for this, or I could just use Celery".

Came here to comment on AGPL. We use Celery a lot. Sometimes for really small projects. Sometimes for Fortune 500 companies. Either way, AGPL isn't an option and managing commercial licensing on something like this for each customer is a huge pain. Despite my technical interest in the project, I can't really get past the licensing. It's a non-starter for me.

I hope it works out for the author. If not, maybe there are other revenue generation options that could be paired with a less restrictive license.

Hopefully more developers stop giving their work away for free so that this attitude can change.

I am curious how the landscape of software development would change if everyone followed suit. I am not sure the effect would be a net positive. $2000 per year is inconvenient. $2000 per year for each and every one of the 20 open source projects you use will be prohibitively expensive. Companies would start writing more of their codes in house instead of using commercial open source software. There would be way fewer freelancers and small software shops as the upfront cost will be enormous. Imagine if you had to pay $10,000 up front to start developing a small commercial Django project (e.g. Django, DRF, Dramatiq, Postgres, Redis).

It will be OK if the commercial option only applied to large organizations. But with the current setup, it just makes Dramatiq unreachable for the small players.

I don't think so. If you were willing to hire a team to implement some solution that existed in OSS form, why wouldn't you just pay for the OSS version, which would likely be cheaper (since you're spreading the price across many different companies). This is already the appeal of OSS - it's cheap, other people build and use it so it's well tested, etc. Those are significant benefits, and worth paying for over an in-house team.

And we already have pricing models for small/medium sized businesses vs large ones. OSS could easily have similar models.

2k USD is somewhere between a man-week and a man-month. Hiring a developer to just reimplement existing solution is prohibitively expensive.

The idea is not that everyone would implement the entire functionality of every library they would otherwise use. With big libraries, each user may need 5% of the functionality. But not the same 5%. If the options were paying thousands of dollars or implementing your own, in many cases the latter would make sense. Definitely not in all cases, I admit.

The way I think about it is what if the only database solution was Oracle. I'm sure many would have wrote their own limited buggy DBMS or used flat files instead. It will certainly be more expensive to write your own Oracle RDBMS from scratch instead of paying for the licence. But you don't need most of the features and for the ones you need, the price point does not make sense for all projects. For some projects, flat files or JSON files saved somewhere on the disk wouldn't sound so bad.

Yep, that's understandable. If someone makes a cost-benefit analysis and decides that they prefer the cheaper option then that's perfectly fine with me, I'll still be here a year later when they realize their mistake. ;)

That said, I do give out commercial licenses for free for companies that are just starting out. My goal with this is not to get rich, it's just to get people to contribute something back in whatever way they can.

I'm sorry about your decision. AGPL is not a way to make corporations pay for your code. It's a way to make corporations stay the hell away from your code. This may be a legitimate goal, but apparently not in your case.

Due to the viral nature of AGPL, they have hard time trying your code to decide if it is worth using. Due to copyright assignment and patent clauses, they have very hard time contributing anything back.

What more or less works is GPL + commercial license, the freemium model. You need to get enough free users on your bandwagon to show corporate users that the code is worth trying, and maybe paying for. You need to get corporations hire you to provide support and develop special features they need in the paid tier.

Not that this problem is completely solved, but likely you heard about e.g. MySQL or Nginx who use the freemium model.

I've always been kind of unclear on how the AGPL works, even after reading various TL;DR-ish explainers. What happens if you link/import AGPL code in, vs running an AGPL service as a separate process as part of your architecture?

I approve the share and share back spirit. AGPL for a library means that the application that uses it must be licensed under the AGPL too, right?

It's a little more nuanced[0] than that and I confess I don't fully understand it myself. IANAL, but my understanding is it depends on how it's used. If a user action on your website ends up triggering Dramatiq code then you have to open source that code in addition to providing access to Dramatiq. If all you do is batch processing on your own servers, then you don't have to do anything; you're merely a user of the software.

[0]: https://opensource.stackexchange.com/a/5004

Is asynchronous interaction still "interaction" for the purposes of the AGPL? If End User A's manual changes are placed into a database or file which is read by Dramatiq code in a subsequent cron-triggered batch process, is that considered interaction? These questions are not at all cut and dry, but companies like MongoDB who use the AGPL could conceivably argue that this would be considered an interaction. [I am not a lawyer.]

As a separate point, it's odd that anyone would use a license that they "don't fully understand," when (presumably) they would hold themselves to a higher standard of understanding the behavior of libraries they uses in their software itself. Is it irrational to consider this, at the very least, an indication of lack of attention to detail, and a tremendous red flag?

I am not a copyright lawyer so when I say I don't fully understand the implications of a particular license that's all I mean. I would say exactly the same of thing of other licenses I've used such as the 3-Clause BSD License, the Apache License and the MIT License. I understand what they're for and broadly how they may be applied, but I don't understand all the intricacies and interactions they have with copyright law because it's not something I have deeply studied. Hope that makes sense!

RabbitMQ is not hard to use directly. I think most developers would be better off doing that, and coming to understand the power of AMQP, rather than adopting cute DSLs like this and Celery that put you into a box and narrow your view of what's possible.

I started that way with Celery and Django, maybe 6 years ago. At one point we found a bug with Celery not resolving 'chord' callbacks when all the parallel tasks had completed. It was difficult to debug, going through Celery's layers of code that try to make various backends present the same interface. We weaned ourselves off Celery and started using just the RabbitMQ and Redis libraries directly. It was definitely a shift worth making, allowing us to make performance/reliability tradeoffs that were better suited to our systems, and opened our eyes to possibilities with RabbitMQ and Redis streams that we hadn't been able to see when looking through a Celery lens.

If there's a place for DSLs like this, it may be when you have very novice programmers needing to write quick throwaway jobs without wanting to spend a lot of time learning the underlying systems. Maybe analysts on a data team, for example.

> I started that way with Celery and Django, maybe 6 years ago. At one point we found a bug with Celery not resolving 'chord' callbacks when all the parallel tasks had completed. It was difficult to debug, going through Celery's layers of code that try to make various backends present the same interface. We weaned ourselves off Celery and started using just the RabbitMQ and Redis libraries directly. It was definitely a shift worth making, allowing us to make performance/reliability tradeoffs that were better suited to our systems, and opened our eyes to possibilities with RabbitMQ and Redis streams that we hadn't been able to see when looking through a Celery lens.

Celery's task workflow "features" have bitten me in the past as well and I agree that it is a complicated piece of software (having had to go through its source code many, many times). That doesn't mean all software in this space has to be like that. That is why one of my goals with Dramatiq is for it to have a very simple and easy to understand core.

> If there's a place for DSLs like this, it may be when you have very novice programmers needing to write quick throwaway jobs without wanting to spend a lot of time learning the underlying systems. Maybe analysts on a data team, for example.

I think you underestimate the amount of value that tools like these bring to the table when it comes to actually shipping a product and getting things done. Of course, you should make an informed decision on the tradeoffs you're making, but calling anyone a "novice" for using an off-the-shelf solution instead of spending valuable time building his/her own is a bit much. :)

I respect the authors of celery and think they've done a good job of making RabbitMQ a bit more accessible over the years. I don't think celery's just for novice users, there are a lot of devs out there who don't really need to invest in learning about their message brokers (short lived projects and non-essential parts of the larger system can easily use celery without needing to worry about it). That said, I really do agree with you on other points.

We ditched celery after many years of use in favor of just connecting to RabbitMQ ourselves. Over the last few years, it's saved a massive amount of time and effort and has made debugging and tracing issues so much easier it's rediculous.

I don't think many long term celery users really appreciate how much time is getting wasted. I understand how it may look like being able to ignore the underlying technology is a time saver, but that time easily gets blown away the second something doesn't behave as it should (and having gone through many of these problems, they're not exactly uncommon).

For anyone curious how much effort it took to write our own consumer/publisher, both are less that 200 lines of code and can be arguably cut down further without losing much (the author liked whitespace etc). Both were written in less than 2 days, including learning time, and are reusable enough to use in other projects.

Seriously? RabbitMQ is a huge waste of complexity for like 95% of modern webapp use cases.

I use tools like Dramatiq specifically because I want to abstract away the complexity of exchanges, bindings, queues, dead lettering, etc...

If you write out all the nouns associated with RMQ you start to realize there is a LOT going on. It’s a very powerful system but convention over configuration doesn’t work here.

No offense, but this sounds a lot like "I can't be bothered to learn about the tech but I want to put it on my CV anyway".

The "95%" of use cases probably don't need a message broker at all. Far too many "modern webapp" developers are shoving message brokers into websites so they can pretend to be more important and sophisticated than they are. If you're not even willing to learn how to connect to RabbitMQ, or any broker, without a big giant messy crutch, I'd suggest you shouldn't use them at all.

Huey: https://huey.readthedocs.io -- even simpler :) redis or sqlite, mit license, greenlet/thread/process worker model support.

I love your work (Peewee is a tool I use very often). I would have liked to use Huey in a recent project but it doesn’t have modular backend support for Rabbit.

Thanks so much. You're right, I think some people have written storage engines for rabbit (as well as mongo) but you might have to search for them. The storage api is simple enough you could write your own in an hour or two.

I might need to do this! And thanks again for your work on Peewee... I think people underestimate it’s power due to the funny name.

I’ve submitted code to this project and just want to state here for the record that Bogdan is very responsive and accepting of contributions. It was a very pleasing experience compared to other open source projects!

I reached out to the Python community recently with the question: “Django is to Flask as Celery is to ______?” Dramatiq was one of the suggested responses and the mission/purpose behind it resonated with me immediately.

I’m a big fan of convention over configuration. Most of the time, if I’m writing software in Python, I don’t want to worry about a system that is not Python. IE, I enjoy an ORM that abstracts PostgreSQL. I enjoy this because it abstracts Rabbit. Obviously you need to understand the underlying system and an ORM is no replacement for knowing the ins and outs of Postgres, just as this is no replacement for understanding Rabbit. But... I love that it does everything I want it to do with minimal or zero configuration.

Nice touch on having prometheus metric support builtin, but in general, if I want a simpler celery, I always use rq[1]. How does this compare to rq? Having this one the website somewhere obvious and why you wrote it would be a really nice touch.

http://python-rq.org/

> Nice touch on having prometheus metric support builtin

Thanks!

> How does this compare to rq? Having this one the website somewhere obvious and why you wrote it would be a really nice touch.

This exists although I suppose I should highlight it better: https://dramatiq.io/motivation.html

I would say the biggest differences to RQ are:

* Dramatiq supports RabbitMQ in addition to Redis,

* Dramatiq uses a mixed multi-process/multi-thread model whereas RQ forks for every task. In my benchmarks, Dramatiq has much, much better throughput than RQ,

* Dramatiq has support for task-level priorities, rate limiting and delayed tasks,

* Dramatiq comes with an in-memory broker you can use for testing.

I figured I'd add a mention of the task queue we use at close.io: https://github.com/closeio/tasktiger

We run several hundred tasktiger workers in production (deployed using Kubernetes) to process the various backend task queues for our service. Project is MIT-licensed if anyone is interested in another celery alternative.

I might have missed it, but is this asyncio-ready?

It currently isn't. I'm holding out until there is some demand for it as I currently don't need it for my own stuff.

Does Dramatiq have first-class support for SQS? That's my biggest gripe with Celery. I mean, it does work with SQS, but it doesn't support it as a result backend, and it doesn't do (eg) batch PUTs automatically, so it's pretty slow.

edit: I search the docs for SQS and got no results, so I'm guessing it's not supported.

It doesn't, but you're the second person to request it. Feel free to open an issue and I'll take a stab at writing an SQS broker. Having looked at it before, it didn't seem like it would take much work at all.

Reply on news.ycombinator.com