Hacker News Reader

Comment by liminal | Hacker News Reader

We just moved from Elastic to Quickwit. Sigh. What other open source, object storage backed logging databases do people recommend?

View on HN

Loki by Grafana Labs is nice (https://grafana.com/oss/loki/). There was a time (3+ years ago) where the product was changing pretty rapidly and much of the documentation was on git, so we had a few headaches doing minor version bumps, but I believe its much more mature now.

Just don’t look at the list of open issues and the lack of responses.

Loki is significantly different in several ways, including the lack of full-text indexing.

Loki has the following problems:

- It is non-trivial to setup and operate. It requires properly configuring and orchestrating numerous components - distributor, ingestor, querier, query frontend, compactor, consul, ruler, memcache, index service, etc. - see https://grafana.com/docs/loki/latest/get-started/architectur...

- Its' configs tend to break with every new release, since some configs become obsolete and some configs change names and/or structure.

- It is very slow on "needle in the haystack" queries such as "search for logs containing some rarely seen trace_id", since it needs to read, unpack and scan all the logs at object storage. This may be also very expensive if the object storage service charges for read IO.

- It doesn't support log fields with big number of unique values (aka high-cardinality log fields). It you'll try storing logs with high-cardinality fields into Loki, then it will quickly explode with enormous RAM usage. Recent Loki releases provide half-baked solution for this problem - "structured metadata" ( https://grafana.com/docs/loki/latest/get-started/labels/stru... ), but it is still experimental and requires non-trivial configs (aka it doesn't work).

I suggest giving SigNoz a try. Built on top of Clickhouse, it’s fast and scalable

https://github.com/SigNoz/signoz

PS: I am maintainer at SigNoz

what another open source you can recommend ?

I'm not the person you asked -- and I also want to be transparent that I only PoC-ed it and due to external circumstances didn't get it all the way out to production -- but I really like how https://github.com/metrico/qryn (AGPLv3) thinks about the world. It is, like SigNoz, unified (logs, metrics, traces) but it actually implements several of the common endpoint schemes allowing it to pretend to be "your favorite tool" which plausibly helps any integration story <https://github.com/metrico/qryn#%EF%B8%8F-query> and <https://github.com/metrico/qryn#-vendors-compatibility>. Had I gotten it up and running, I was going to contribute a Splunk ingest adapter since that was what we were trying to replace

I was going to take advantage of Clickhouse using S3 as warm-to-cold storage since my mental model is that most logs, metrics, and traces are written and not read https://clickhouse.com/docs/en/integrations/s3#configuring-s...

I believe one could do that with SigNoz, too, so I don't mean to imply that trickery was qryn specific, just that I didn't want to get into the "constantly resizing io3 PVC" game

qryn developer here, thanks for the mention and for the kind words! hope we'll get another shot in the future!

Maybe Loki, but it has some limitations related to high cardinality data and indices (https://signoz.io/blog/logs-performance-benchmark/?utm_sourc...)

Take a look at VictoriaLogs - https://docs.victoriametrics.com/victorialogs/ . It is a single small executable, which runs out of the box with the default configs (e.g. it is zero-config) and stores all the data into a single directory (support for object storage coming soon).

Reply on news.ycombinator.com