Hacker News Reader

Comment by evgen | Hacker News Reader

The throughput numbers on the chart would have seemed a bit more valid if the task had not included stuffing the result into a protobuf given the notoriously slow performance of the standard python protobuf implementation that would have been used by Celery and PyInvoke. If they wanted to impress, or at least be honest, the data would have stayed in JSON the whole way through the stack and the worker task would have been a simple string manipulation of something similar.

View on HN

Hi there,

The task benchmarked was from a component of our messaging stack at work that I'd ported. My intention was to offer an example of the end-to-end performance of a real-world task that someone might be writing, rather than a sample task that's not much more than a no-op. I'd suggest that this particular one is real-world as it's straight out of one of our applications. I've no intention of trying to be dishonest here - just offering the measurements of my (and our) internal evaluation of the tool for our needs.

While I can't provide the source of the task, I can offer the quick-and-dirty source of the quick little "PyInvoker" I'd whipped up (sorry - didn't realize there was something called PyInvoke at the time). It just takes a message, unpacks the JSON, and uses getattr to call the appropriate task. Nothing fancy like e-mail error notifications, retries, and the like: https://gist.github.com/18a30689832569d67861

Anyhow, always and absolutely take any claims regarding the performance of a tool with a massive grain of salt, and try them for yourself to see if they suit your needs. Octobot's not designed to replace background processing in most applications as tools like Celery and DelayedJob (both of which I use myself) are great, a bit easier to write for, and a bit simpler to get up and running, depending on the application and language being used.

There's no intention to have slighted anyone or any other project here. I'm stoked that a lot of tools exist in this space. I just hadn't seen one on the JVM that offered this level of simplicity and parallelism with a bit of restraint when it comes to feature creep. But if you have an application that demands high throughput / low latency execution of tasks, this might be worth evaluating.

No problem. I did not want to suggest that you were fudging numbers or anything, but it just appeared to be a strange selection of parameters for a comparison chart (and to be honest, whenever I see a comparison chart that shows X outperforming Y by more than 10X I tend to take a closer look at the details of what is being compared...)

As to the task selection, I do not think showing a "real-world" task help here, since we all have different tasks to be performed out in the real world. I want to know what overhead a particular piece of infrastructure adds, so something as close as possible to a no-op is a useful data point.

I'm interested in seing the code/config used to benchark Celery. The default settings are not at all optimized for processing lots of small jobs, and you could easily tweak it to get a 100x speed up for that use case, e.g.:

   CELERYD_PREFETCH_MULTIPLIER = 0
   CELERY_DISABLE_RATE_LIMITS = True

Also, channels are not re-used unless you explicitly pass the Publisher, so e.g.

   publisher = task.get_publisher()
   for i in xrange(1000):
       task.apply_async(args=(i, ), publisher=publisher)
   publisher.close()

is known to be a massive speed-up for sending tasks in batch (it seems the creation of channels is very expensive in pyamqplib).

By the way the performance increase you're seeing with the PyInvoker (from your gist) is most likely because it doesn't have prefetch_count enabled.

Celery enables this so a single worker doesn't suck in a million messages at a time, and to balance the work load between available resources. As noted previously it can be disabled.

Btw, octobot looks great, maybe we can share ideas.

Right on, thanks Ask! I'm checking out some of this right now and might not be able to get through it all today, but will give it a try. Just shot you a couple messages outside of HN - love to talk when you have a chance!

Does CELERYD_PREFETCH_MULTIPLIER do what I think it does (disable prefetching so workers get items as they need them)? I like Celery a lot, having used it for a few days, but the documentation is a bit frustrating :/

Reply on news.ycombinator.com