Hacker News Reader

Comment by notdarkyet | Hacker News Reader

Do you have additional examples of those simpler ways? I totally understand how Celery can be a hammer and everything is a nail type situation.

View on HN

I am a big fan of Huey. And they have a Django module https://huey.readthedocs.io/en/latest/

I would just use serverless functions to achieve the same thing personally, if you're in the cloud already, chances are high you can trigger functions based on new records in a database or new file uploaded, or what have you. Then you don't need to import much outside of the serverless SDK which should typically be pretty minimal.

That's how I did a timed function for a Django project we were hosting in Azure anyway.

Serverless functions are good for some stuff: clear this cache on this schedule, run this task when XYZ happens, etc..

But that is not the popular usecase for celery. Often you want "some code" (that you likely already have written) to be executed async. Sure, you can create a public interface for "some code" then write a record, that the serverless function is looking for, that then calls back to that interface (but is it a web interface???? then you have a problem where if the job takes too long to complete, what about http timeouts ... ) and now you're really creating a big circle for something that should be simple: execute some code outside of the request (send an email, hit an api, whatever).

Serverless functions really shouldn't contain much logic either, because it's too complicated to test.

Ive used serverless in this way as well when it was a long running process, basically the end-user needed to upload a Shapefile, and some of them can be quite large, so I kicked off an Azure Function once the file was uploaded to parse the file in the background. If it's something that will halt the browser when it needs to run in the background instead, that's where I'll use Serverless if it makes sense. I'm not fond of having my web server running things in the background it ruins my mental model of the web.

> I'm not fond of having my web server running things in the background it ruins my mental model of the web.

Well yes, and you run into other problems as well (now if your process dies or you deploy or something you have to be careful to not kill running jobs).

How much of your logic is in the azure function?

Only things that needed to be done behind the scenes that could take a long time, like we would take JTIFFs that could be gigs of data, and analyze them and create variations of the same image. You don't want someone who just uploaded a 1GB file waiting for you to also analyze it as well.

Eh, yeah, I just don't like any real logic being in those types of systems. Just too hard to test and chase down issues.

I guess "convert this image" is ok, since it is kind of a "pure function" sorta.

Correct, I only use them for very specific needs where it would slow down my website, or be weird to spin off a new thread in the background. I do know Azure supports WebJobs which is probably the closest you can get to something like Celery, which funnily enough, is what Azure Functions (at least originally) are built on top of.

https://github.com/dabapps/django-db-queue

(I am the author)

EDIT: oh hi, Andy :)

Since everyone suggests alternatives to Celery, may I plug my own healthier celery?

https://github.com/NicolasLM/spinach

Reply on news.ycombinator.com