Replacing Celery with Elixir in a Python Project

> The slides mention problems with Celery and RAM usage when writing crawlers, but since this is a mostly I/O-bound task you should be using the eventlet/gevent execution pools instead of the multiprocessing one.

Here: https://klibert.pl/statics/python-and-elixir/#/5/6 in line 18 you can see a `_do_some_real_processing_function`. The whole premise of the project is that this function is CPU-bound. My processor has four cores, so I create a pool of 4 Python processes (https://klibert.pl/statics/python-and-elixir/#/5/2 line 10).

Fetching pages is an IO-bound task, so it's done by Elixir. There we have a pool (for rate limiting) of 10 processes (the Erlang ones - important distinction) that do the downloading.

I think the closest analogy to what happens in this project is a Twisted (EDIT: or any other concurrent, but not parallel framework) app which uses a pool of processes for CPU-bound tasks. Here the Twisted part is replaced with Elixir.

EDIT: Also, we use Celery at work extensively and it works great and there is no real need to replace it with anything! Again, this project is just a tech demo, it doesn't make (much) sense on it's own. But there are other possible integration patterns where Elixir and Python have different roles, which actually do make sense. I think.

as a viewer you can just press space for the traditional progression.

(And the "you idiot, you weren't using the correct navigation keys" is not a particularly satisfying answer, I navigated with left and right arrows the way I have always navigated slides for years.)

By "replacing" I mean pretty literally getting the same effect as with Celery. [EDIT: BTW, in the recent post about Celery it was praised because it lets you compose background/async tasks. Of course, Erlang has something like that, too: https://chrisb.host.cs.st-andrews.ac.uk/skel-test-master/tut...] There really is no other reliable way around the GIL in Python other than multiprocessing.

The main idea here is that we should keep IO-bound code on the Elixir side as it's simply more efficient and easier to write there, but to delegate CPU-bound code to a pool of external processes. But this is only one of many possible patterns of integration: I can imagine a Django project where most of the logic is on the Python side and Elixir only handles WebSockets/long polling connections. Thanks to ErlPort passing data and calling functions between Elixir and Python is effortless, you can call an Elixir function just as easily as you can make Elixir call a Python function in a different process.

Moreover, ErlPort supports Ruby, so you can have workers written in it, too. There's no problem with workers in other languages, too, as long as you write the glue code yourself (or generate using Elixir macros).

[1] It bears repeating: Erlang processes are not OS-level processes. Erlang Virtual Machine, BEAM, runs in a single OS-level process. Erlang processes are closer to green-threads or Tasklets as known in Stackless Python. They are extremely lightweight, implicitly scheduled user-space tasks, which share no memory. Erlang schedules its processes on a pool of OS-level threads for optimal utilization of CPU cores, but this is an implementation detail. What's important is that Erlang processes are providing isolation in terms of memory used and error handling, just like OS-level processes. Conceptually both kinds of processes are very similar, but their implementations are nothing alike.

That's right. It wasn't supposed to be a real project, it's a just a tech demo of sorts. I will have some more real-world uses for the integration in my Raspberry project, I think, but I'm not there yet.

Actually, that's how ErlPort works. You start a Python process and then you call some functions inside that process. How many functions you'd like to run using a single Python process is up to you - you can spawn a new worker for every call or spawn only one worker and stick to it unless it's killed somehow.

You can also register callbacks on either side (Elixir or Python) and with a bit more effort you can make Python process accept normal Elixir messages and answer in the same way.

> How do web frameworks like Flask handle multiple concurrent requests?

In my experience, they mostly rely on uWSGI and a pool of processes...

The point is that the integration Python<->Elixir can be so tight and that there's little overhead when using each language for things it's good at.

However, there are things like generating PDFs with graphs based on the tabular data on the pages or running some more involved Pandas TimeSeries transformations which are simply not available in Elixir. Nor they should be, I think: reportlab or Pandas are already written and do a good job at what they are meant to do. This is the idea: we write a crawler in Elixir and delegate processing to something else. Anything else, in fact: Python was chosen because of ErlPort and how easy it is to integrate, but your workers can in practice be written in anything that understands JSON.

It was stable when I used it. But it's also rather simple and short (less than ~1k loc on Python side, for example), so I think it wouldn't be a heavy burden to maintain, even if the current maintainers and users all dropped dead tomorrow :-)