evanxg852000

(disclaimer: author here) I have haven't used Erlang/Elixir/OTP so I won't be able to answer to your question.

The goal of the framework is to build a data processing pipeline with relatively big tasks, we only have to process tens or hundreds of messages per seconds. Some are CPU intensive and can last 30 seconds. Other tasks are IO intensive.

From my little knowledge of Loki's internals. I think contrary to Loki, Quickwit uses a fully feature search engine library underneath called Tantivy (https://github.com/quickwit-oss/tantivy). Quickwit offers different services (indexer, searcher, ..) that can be ran and scaled independently. It also supports indexing from various sources including file, Rest API, Kafka, Pulsar, Kinesis and more are planned based on community interest. Last but not least, Elasticsearch query API support is being worked on.

Thanks a lot for your kind comment!

To complete your description of Quickwit, It is a distributed search engine for logs and traces. It's written in Rust, ingest at speed, horizontally scalable, and separates compute from storage.

Last but not least, Grafana integration is planned for next month :)

I forgot to reference the issue https://github.com/quickwit-oss/chitchat/issues/30

No it's not.

Thanks a lot, you can always file a issue on https://github.com/quickwit-oss/chitchat. We can improve the documentation, add more example and mostly learn from your use case.

Depending on the need, it can be implemented on client side as well as on chitchat. We actually thought of this feature as a way for a node to normaly exclude itself from the cluster (maybe for application update). We are still evaluating how necessary this is because a node crash or normal shutdown can just work. Future experience will surely tell us.

Thanks a lot.

- In term of message overhead there is not much difference. one details is that each node keep incrementing his heartbeat counter continuously, making it a state change that need to be propagated.

- slower here depends on the number of node selected to gossip with. if you choose a long gossip period yes, but in real system the gossip period is short enough to not notice it. So slower means compared to Rumor-mongering style. Not as in having an inconsistent cluster state.

-(Thanks) I will probably amend this as a note: A seed node can be any node in the cluster we know is reliable and will almost always be there; kind of like the ACE within your cluster, they can be many.

-Garbage collection :"... We mitigate this by periodically garbage collecting obsolete nodes."

Awesome!!!

Glad to hear your interest, we have a list of sources we want to support here https://github.com/quickwit-inc/quickwit/issues/1000 You can manifest your interest, this will help us prioritise in our roadmap.