Like said in one comment here, it works well on billions of logs on one modest instance. And Grafana integration is on the way :)
https://github.com/quickwit-oss/quickwit
(disclaimer: I'm one of the cofounders)
Like said in one comment here, it works well on billions of logs on one modest instance. And Grafana integration is on the way :)
https://github.com/quickwit-oss/quickwit
(disclaimer: I'm one of the cofounders)
Unlike index-free solutions like Loki or Parseable, Quickwit is built on top of a modern full-text search index (tantivy). At query time, Quickwit produces much faster results (all other things being equal: CPU, memory, etc.), especially when the volume of data to analyze is large or queries are complex (high cardinality values, aggregations). Quickwit also stores data in a columnar format, so it's also good at OLAP-style queries (no joins though).
This comes with a cost during ingestion; Quickwit is more resource hungry than Loki but can still ingest at 20MB/s to 40 MB/s on a commodity instance with 4CPU. Similarly, regarding storage footprint, Loki compresses logs better because it does not maintain those extra data structures. Still, a Quickwit index tends to be much smaller than an Elasticsearch index.
The next release of Quickwit (may) will be shortly followed by the publication of a benchmark against Elasticsearch/OpenSearch, and by another one later, against Loki. You'll be able to see for yourself.