I came here to say something similar. It has also occurred to me that vectordev hasn’t gotten much attention here on HN. It’s a great tool. You can listen on a TCP socket, write to Kafka in an upstream instance, then read from Kafka, transform data, and write to ES or whatever downstream. I’ve even recently completed tests where I could write from vector to vector at over 6 million records a second.

I don’t really mean to discourage the author of the post, I just realize this can be done with two simple vector configs.

In particular Vector can do simple transforms on the messages (or quite complex transforms using Lua script), and insert directly into Clickhouse.

I respect the simplicity of Redis here, but personally I would consider Kafka if I wanted to scale this setup further, maybe one of:

Vector -> Kafka -> Clickhouse (using Clickhouse's built-in Kafka support)

Vector -> Kafka -> Vector -> Clickhouse

This also lets you stream the logs live if you want to.

How good are they at opentelemetry support?

I like FluentBit, written in C and very light on resources.