VictoriaLogs 0.1.0:log management and log analytics system from VictoriaMetrics | Hacker News Reader

VictoriaLogs 0.1.0:log management and log analytics system from VictoriaMetrics

docs.victoriametrics.com

28 pointsyla927 comments

VictoriaLogs core developer here.

The main motivation for creating VictoriaLogs was not so good experience with the existing open source solutions for logs - mainly ElasticSearch (or Opensearch) and Grafana Loki. These systems have the following annoying issues, which make users unhappy:

- Non-trivial setup and operation

- Hard to use query language

- Too high resource usage (CPU, RAM, disk space, disk IO)

- Bad integration with the traditional command-line tools for logs analysis - grep, head, less, jq, awk, etc.

So we at VictoriaMetrics decided creating user-friendly open source database for logs - VictoriaLogs. We believe it solves pain points of Elasticsearch and Grafana Loki mentioned above.

Try it on your production workloads and compare how it feels to your current solution for logs. https://docs.victoriametrics.com/VictoriaLogs/

Since you're here, I would really enjoy reading about the operational concerns about running V.L. for real (and my apologies for not digging into the repo for "code is the truth" type docs):

- Does it use its own flatfile system, or some novel storage mechanism?

- Any plans on offering "S3-ish" store for practically unlimited storage (at, of course, the risk of latency for fetches)?

- Does it tolerate running on NFS?

- VictoriaLogs uses its own storage format, which is inspired by ClickHouse storage format [1]. This is basically a column storage format optimized for structured logs with arbitrary number of fields.

- Yes, VictoriaLogs is going to support object storage. See our roadmap [2]

- VictoriaLogs should work good on NFS. It writes data to disk in big sequential chunks, which should work optimally on NFS. It reads data in parallel from multiple threads during queries. This should help increasing query performance on high-latency storage such as NFS.

[1] https://clickhouse.com/docs/en/development/architecture

[2] https://docs.victoriametrics.com/VictoriaLogs/Roadmap.html

damn shame about using "!" in place of the (IMHO) far more common "-" syntax: https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html#lo...

Initially LogsQL used the '-' char as 'NOT' shortcut. But lately we decided replacing it with '!' char, since the '-' is frequently used in full-text search phrases and/or field names. For example: 'some-field:some-value'. This could bring more confusion to LogsQL. So we decided to use '!' char as 'NOT' shortcut, since it occurs less frequently than '-' in typical qieries.

I mean, it's your QL so you can do as you see fit but surely `_msg:alpha-beta` didn't previously parse as `{"_msg":["alpha", "NOT", "beta"]}` and nor would I expect `alpha-beta:charlie` to split in such a non-whitespace way

I guess it boils down to whether LogsQL seeks to separate itself from similar solutions, in which case doing abnormal syntax is great, because there's no "uncanny search syntax valley" to be found, or whether it seeks to ease transition from existing search syntaxes in which case every deviation is one more "oh, right" to keep in one's head while transitioning between the systems

- https://lucene.apache.org/core/9_6_0/queryparser/org/apache/...

- https://support.google.com/websearch/answer/2466433

- https://slack.com/help/articles/202528808-Search-in-Slack#na...

likely a bunch more. OTOH, maybe this is sampling bias since I'm sure there are syntaxes out there which use bang negation, too, I just haven't used them as much

Fair reasoning. We need to think more about using the '-' instead of '!' in LogsQL.

On the other hand, the '!' char is used in almost every programming language as 'NOT' operator. Probably, this is my professional bias.

Reply on news.ycombinator.com