Why do we need Flask, Celery, and Redis? (2019)

Forget email, say you have a app that scans links in comments for maliciousness. You rely on an internal api for checking against a known blacklist, which follows shortened links first, and an external api from a third party. You want the comment to appear to submit instantly to the poster but are comfortable waiting for it to appear for everyone else. What are your options?

You could certainly use message queues and workers. If you’re cloud native maybe you leverage lambdas. Maybe you spin up an independent service that does the processing and inserting into the database in the background, and all you need to do is send a simple HTTP request on an internal network.

Your solution depends on your throughout requirements, the size of your team and their engineering capabilities, what existing solutions you have in place. Everything has its pros and cons. Pretending that celery/redis is useless and would be solved if everyone just used Java ignores the fact that celery and redis are widely popular and drive many successful applications and use cases.

If you don't need all that, then it's not a problem in Python either. You don't even need asyncio, you can just use a ThreadPool or a ProcessPool, dump stuff with pickle/shelve/sqlite3 and be on your way.

It's not in the standard library, and there are probably other options too now (not a heavy Python user any more), but Python has had easy parallelism for at least a decade (probably longer).

You can't say that when Python shoves Pip/Poetry/VirtualEnv/Black/Flake and what not into people. In contrast, Go has built-in package management and gofmt. Python is essentially gate-keeping new devs.

Can you provide some context for this statement? I've used Python asyncio extensively in Fargate (no ASGI frontend), and the developer experience is far from Go; however, I don't see how an ASGI framework can fix this. It seems like it offers the same general course-grained parallelism that you get from a containerized environment like Fargate except that Fargate abstracts over a cluster while ASGI frameworks presumably just abstract over a handful of CPUs.

For example, we have a large data structure that we have to load and process for each request. We want to parallelize the processing of that structure, but the costs to pickle it for a multiprocessing approach are much too large. We've considered the memory mapped file approach, but it has its own issues. We're also looking at stateful-server solutions, like Dask, but now we're talking about running infrastructure. In Go, we could just fork a few goroutines and be on our way.

Of course if you are stuck with Python it's better than nothing.

(1) it's undersupported (e.g. you can in theory download s3 files using async botocore, but in practice it is hard to use because of strict botocore version dependencies)

(2) it isn't natural - once you're in the event loop it makes sense, but using an event loop alongside normal python code is confusing at best.

(3) it got introduced too late. The best async primitives are only available in pretty recent versions of python3.

The difference with go is that it go has these primitives built in from the beginning and using them doesn't introduce interoperability problems.

This is also why McDonald's introduced table service, which is only in restaurants that have a layout where it's impossible to hide how many people are waiting. It costs manpower to deliver food to tables, but the additional orders are worth it.

McD's don't mind if you have to wait, they mind if you leave before you order. "Busy-looking queue" is a much more frequent problem than "totally-packed-restaurant".

Source: I like hamburgers + I geek out over stuff like this. I.e., Just Guessing.

The kiosk system is human-less thinking, we're not machines operating with pure logistics in mind. We like to have responsibilities in a way. When I'm talking to a cashier, he/she feels a duty to do something from A to B. With decoupled ordering.. nobody knows who I am, nobody really cares (McDonalds doesn't pay nor train for welcoming and service mindset). I'm just a thing carrying a ticket.

I've seen things so absurd, people walking around not knowing who to talk to, where to go, what to assume; both customers or employees. It was a surrealist comical situation.

Also my average time to serve for 1 hamburger with no other customer around is 7min. (5-15). Talk about fast food :)

Another point, I don't mean to make people work more, but I even prefer busy waiting lines with hectic kitchen action. That's what McDonalds were, high throughput grill. It felt something.. now it's all dull and clinic.

Oh and lastly .. the kiosk are fugly. They break the room space, break the flow of people, are way too big for their purpose (100$ a stupid tiny 80s monochrome terminal, would do better :p)

ps: This decoupling idea was tried at a company I worked for. Exact same principle (which I also gave some credit at first), split everything in small chunks so people can go faster.. it all went worse because nobody took responsibility for anything since a single task was now a dozen tiny bits done by a dozen people not really knowing what their bit was for. They just passed the products from hand to hand, not being able to track who or what was wrong until the last guy received all the shit because he's the one to show the result to the managers :)

Questions like: should I send this email in-line in the web request? Get a very easy answer: no, just stick do it later. Sure, sending an email is probably fine to do in-line for now, but months in you may realise that things are slow, that you’re sending emails and rolling back transactions later, or committing the transaction but losing the email that needed to be sent, or all manner of other annoying edge cases. Queues don’t solve everything, but they can be an ok answer to a lot of stuff for a long time.

For basic sites, yeah maybe not necessary, but a reliable background processing system has always been a significant accelerator in my projects.

Several developers like to overengineer and "go for celery" (also applies to other technologies with other uses) even for small things.

You don't need Celery to run a batch job every day for example. Or to even do some parallel processing.

"Oh but python multithreading sucks" do you know when it does not suck? When your thread is waiting on something else. Also there's the multiprocessing module with a lot of "batteries included" for basic use cases.

Not to mention (my biggest pet-peeve of Celery) is that it "forces" you to work with a task queue model. Not a data queue model. Sure, it helps a lot when you need that, but sometimes you just need a queue.

For example, sending a mail can take a while because mailservers have queues and whatnot. If you send a mail after signing up a user for them to verify their email, you don't have to wait until the mail is "really" sent before letting them know that their signup has been processed. You can tell a background worker to send the mail and return to the user much faster. For another example, in a previous job I worked for a big file sharing service. If user wanted their files deleted that caused all sorts of calls to AWS to actually delete the files, which could take a while. However, from the user perspective it was pretty fast because all we had to do was set the file in the database to "delete in progress" state and tell a background worker to delete the files. Then we could show the user that their files were being deleted within a couple dozen milliseconds instead of having to wait for all the AWS calls to complete.

Although this delay chain might be considered worth it you don't want to scale the frontend to multiple workers for some reason e.g. single-threaded evented runtime, or GIL runtime (python, ocaml), or if you want to avoid CPU-hard tasks being executed on your frontends.

In that case, it might be valuable to transform CPU waits into IO waits by moving the CPU work to a jobs queue, possibly running its workers on a different set of machines entirely.

Really though, I think a lot of people use celery for offloading things like email sending and API calls which, IMHO, isn’t really worth the complexity (especially as SMTP is basically a queue anyway). Of course, YMMV depending on your use case.

However, I find it is often more worthwhile for:

1. Tasks which take a long time to run

2. Tasks which need to happen on a schedule, rather than in response to a user request.

There is an option 3 too, which is for inter-software communication. Eg events or RPCs, but I found Celery to be very much a square-peg-round-hole for this, which is why I developed Lightbus (http://lightbus.org). Lightbus also supports background tasks and scheduled tasks. /plug

But as we decided to implement an MVP with a single HTTP request from the client, this whole separation doesn't make any sense, exactly as you noted.

The correct pattern should be a client submits a request to get something done to a thin layer, receives a ticket that allows it to claim the result and goes away to either check for the result via polling for the ticket or receives a call back.

It depends. If you needed a coordination layer or you needed to isolate certain types of traffic then it makes more sense. I assume your alternative here is "why not just have a new client tier doing work" which is a reasonable architecture too

> One aspect of this set up I’ve never been able to understand is how the application then gets the result from the worker?

Often they don't in these architectures. I found it a little strange that a job queue was being used to serve (what seems like) synchronous traffic. Usually I see job queues in the wild used for async/send-and-forget workloads

Personally I would rather chain multiple synchronous service calls if I needed a synchronous workflow. It's just simpler to me to stick web workers behind a load balancer and scale that. This is less elegant when things take a very long time though or are prone to retries. Either the client or server needs to be responsible for queuing/message persistence/retries - with services the client does it, with job queues the server does it

What happens when you want to retry jobs with exponential back off, or rate limit a task, or track completed / failed jobs?

You can wire all of this stuff up yourself but it's a hugely complicated problem and a massive time sink, but Celery gives you this stuff out of the box. With a decorator or 2 you can do all of those things on any tasks you want.

I use it in pretty much every Flask project, even on single box deploys.

In Python too:

    # or ThreadPoolExecutor depending of the type of work
    from concurrent.futures import ProcessPoolExecutor, as_completed

    with ProcessPoolExecutor(max_workers=2) as executor:

        a = executor.submit(any_function)
        b = executor.submit(any_function)

        for future in as_completed((a, b)):
            print(future.result())

No, the real reason we use celery is that it solves many problems at once:

  - it features configurable task queues with priorities
 
  - it comes as an independent service accessible from multiple processes

  - its activity can be inspected and monitored with stuff like flower

  - it has the ability to create persistent queues that survive restart

  - error handling is backed in, they are logged and won't break your process
 
  - it offers an optional result backend for persisting results and errors
 
  - many distribution and error handling strategies can be configured

  - tasks are composable, can depend on each others or be grouped

  - celery also does recurring tasks, and better than cron

  - you can start tasks from other languages than Python

It is true that Python concurrency story is not comparable to something like Go, Erlang or Rust, but very common use cases are solved.

In the same way, we can perfectly use shelves instead of redis.

We use redis beacuse it solves many problems at once:

    - it's very performant on a single machine, and can be load balanced if needed

    - it has expiration baked in

    - it offers many powerful data structures 

    - it embeds numerous strategies to deal with persistence and resilience

    - it's accessible from multiple processes, and multiple languages

    - its ecosystem is great, and it's hugely versatile

Once you are at it, why not later use it for other stuff, like storing the result of background tasks, log stream, geographical pin pointing, hyperlolog stats, etc. ? There are so many things it does better than your DBMS, faster, easier or for less resources.

It's such fantastic software really.

But no, nothing prevents you in Python to create a queue manually and serialize it manually. It's just more work for less features.

You have an operating system that you can use. You don't really _need_ concurrency when you have a machine that can timeshare amongst processes. That's the world Python was designed for

> In most other languages you can get away with just running tasks in the background for a really long time before you need spin up a distributed task queue

This is partly true, but not a lot of people do this because you need to persist those tasks unless you want them dropped during a reboot. Same logic goes for entire machines disappearing. You _can_ get by without a distributed system, but you will need to tolerate loss in those scenarios. Those losses are non-trivial for most apps/companies, so it doesn't seem all that practical to me to consider a world without a distributed system (and yes, persisting things in postgres/mysql before they are being worked on is still using a distributed system)

They do, it's a WSGI application. Flask has been multithreaded for many years. Python also has multi-process queues that don't need an extra process.