VictoriaLogs: A Grafana Dashboard for AWS VPC Flow Logs – Migrating from Grafan | Hacker News Reader

VictoriaLogs: A Grafana Dashboard for AWS VPC Flow Logs – Migrating from Grafan

rtfm.co.ua

49 pointstodsacerdoti9 comments

What’s the price difference?

VictoriaLogs author here

Both Loki and VictoriaLogs are open source and free to use. But they have non-zero costs for the hardware they use and non-zero maintenance and operational costs.

VictoriaLogs is much easier to configure and run properly than Grafana Loki. VictoriaLogs runs smoothly with default configuration. It consists of a single small executable, which stores the ingested logs to the local filesystem. It doesn't break during upgrades because of good backwards compatibility.

Loki, on the other side, requires non-trivial configuration and tuning to run properly. This configuration breaks frequently during upgrades. Loki consists of many microservices, which aren't easy to configure and orchestrate. It stores data to S3 (or any other S3-compatible object storage). This brings an additional dependency, which isn't needed in most practical setups.

While S3 storage is usually cheaper than block storage, the price difference isn't so big (2x) when using HDD-based block storage (VictoriaLogs is optimized for HDD). Also, IO operations at S3 usually requires additional costs, which are frequently overlooked when comparing S3 costs to block storage costs.

Also, VictoriaLogs runs typical queries over logs at much faster speed than Grafana Loki. This is clearly shown in the linked article.

Are there any stats including price? Say something like a TB of logs a day, retention for a year? Obviously the older they get, the less likely they are to ever be looked at.

The costs highly depend on the shape of the ingested logs and the type of queries, which are performed over the ingested logs.

For example, VictoriaLogs compresses typical logs from Kubernetes containers by up to 50 times. This means it needs 20GB of persistent storage for saving 1 terabyte of Kubernetes logs. A terabyte of HDD-based persistent storage cost $40/month at Google Cloud. This means you can keep 50TB of Kubernetes logs in VictoriaMetrics at $40/month. Additionally, you'll need to pay a few hundred bucks per month for CPU and RAM for efficient data ingestion and querying.

Rather than promtail, have you looked at Vector? I just started an MVP of shipping logs directly to our product for end-user consumption using Vector + NATS, and it’s a killer combination.

Compared to Promtail, you have incredibly flexibility with how you collect, transform, and ship logs, with end to end confirmation and great documentation. Using promtail felt inferior once I made the switch

https://vector.dev/

https://docs.nats.io/

Whoops, thought you were OP

Agreed that Vector is much better for log collection, transformation and forwarding to the log storage than Promtail. VictoriaLogs can accept logs from Vector - https://docs.victoriametrics.com/victorialogs/data-ingestion...

Hey, I heard you liked JSON?

JSON is the universal format for structured logs. It is supported by all the popular log collectors and shippers. It is natively supported by VictoriaLogs [1] together with other popular data ingestion formats for logs [2].

[1] https://docs.victoriametrics.com/victorialogs/keyconcepts/#d...

[2] https://docs.victoriametrics.com/victorialogs/data-ingestion...

Reply on news.ycombinator.com