Efficient indexing with Quickwit Rust actor framework | Hacker News Reader

Efficient indexing with Quickwit Rust actor framework

quickwit.io

How does this compare to Erlang/Elixir? Is any gain of using a strictly typed language or the benefits are mostly related to Rust's CPU efficiency?

(disclaimer: author here) I have haven't used Erlang/Elixir/OTP so I won't be able to answer to your question.

The goal of the framework is to build a data processing pipeline with relatively big tasks, we only have to process tens or hundreds of messages per seconds. Some are CPU intensive and can last 30 seconds. Other tasks are IO intensive.

how much actors is possible to spawn on usual hardware?

Based on the descriptions of the tasks here, they're going to run out of CPU for the actual processing tasks long before they run out of CPU for the actor management. So it doesn't matter whether their actor implementation is capable of spawning 25 million actors doing nothing on commodity hardware when they can probably only keep maybe 2 times or 3 times the number of CPU's worth of actors busy anyhow.

This is indeed correct. Yes, we have comparatively few actors. One thing that is not explained in the blog post however, is that we can run several indexing pipeline on the same machine. We have a bunch of tricks in our framework to run ~1000ish index pipeline on the same server. That's a target of around 10k actors in the most extreme case.

We also send very few messages: the smallest message we send is represent batches of log lines.

As a result, our framework does not really care about performances. On one thread, it can process ~2.5 millions messages on my laptop: this is well above our needs, but it is correct to say that our framework is very slow compared to others.

Also could these be combined with e.g. rustler, having broadway in elixir to do the pipelining while rust NIF does the computationally intensive /cpu bound parts?

I don't know rustler limitations.

Let me just tell you the limitations on Quickwit side: the framework is very bound to Quickwit code base right now and uses tokio runtimes to run the different kind of tasks.

We put the MIT licence so people can copy/paste pieces of code.

any plans on migrating it to its own repo? presumably when it's more mature?

You guessed it :)

It's not mature enough and it will require some work to migrate it as we would need to clean some Quickwit dependencies. It was just easier to add the MIT licence directly in Quickwit directory to allow anyone to copy/paste the code.

Reply on news.ycombinator.com