Ask HN: Logstash vs. Kafka? | Hacker News Reader

Ask HN: Logstash vs. Kafka?

3 pointsalbertlie6 comments

Hi all,

I'm trying to build realtime data infrastructure for logging. For ingestion layer, I'm thinking about using Kafka / Logstash for logging ingestion layer and then after that I can store it in any database and I can easily change the store without changing the ingestion layer in the future.

Any experience using logstash or Kafka in production?

Also, additional question is, I concern a lot for missing data case when we're sending data to logstash or kafka using kind of lightweight shipper like filebeat? Any experience for handling missing data at scale?

View on HN

Are you really ready to forego hundreds of megabytes of RAM for merely log shipping? Fluentd could be a cheaper alternative to JVM-based routers.

Also, what exactly is your question?

> Any experience for handling missing data at scale?

For logs? Missing logs don't matter (unless they're a required audit data). Your system should be prepared not to fall apart on missing hours or days of logs, similar to how it should treat missing metrics and other monitoring data.

And what volume is "at scale"?

Hi Dozzie,

Thanks for the answer.

Just for clarification,

1. For my case (yea, they are kind of required audit data), I need the log to be able to not loss when sending to logstash or kafka, probably this should be handled in client side by buffering the data before sending to logstash?

2. For volume "at scale", my point is just for clarify whether this logstash could be good to be used for production for big volumes. Maybe can be like 100GB per day

> 1. For my case (yea, they are kind of required audit data)

What kind and required by whom? Most of such things are "if you have all, good, if half an hour is missing, we'll do without those", not something akin to transaction data in relational database where you cannot lose even a single byte.

> I need the log to be able to not loss when sending to logstash or kafka

What is "not to lose logs"? Is losing a single record OK? Is losing 1% of logs OK? How long network problems should be tolerated? What to do if such network problems are longer than anticipated and disk buffer is running out of space?

You have said virtually nothing about what your logs consist of and what kind of reliability you require from them.

Hi Dozzie,

That's great question

> What kind and required by whom? Most of such things are "if you have all, good, if half an hour is missing, we'll do without those", not something akin to transaction data in relational database where you cannot lose even a single byte.

So we will do log analysis from the data (counting, sum of the amount from the data, etc). So ideally need 100% precision and we will do the analysis in daily basis

> What is "not to lose logs"? Is losing a single record OK? Is losing 1% of logs OK? How long network problems should be tolerated? What to do if such network problems are longer than anticipated and disk buffer is running out of space?

In my case, losing single record is significant enough because we will do calculation for those logs. For the network tolerance, I think that's very good question, I'll take that as my consideration too. But the network problems that I mean before is the intermittent one like < 1 minutes and with assumption the data buffered is not until making the server running out of space

Also, In terms of reliability, which one is better between logstash and fluentd?

I saw logstash and fluentd is really similar, The difference that I found is just fluentd has buffering into disk mechanism, but logstash not (logstash only has the in memory queue buffering)

We're using Kafka as a log delivery platform and are quite happy with it. Kafka by nature is highly available and can be scaled quite trivially with the log load by adding new cluster nodes.

We've decided to use journald for storing all of our application data. We pump the entries from journald to Kafka by using a tool that we open sourced: https://github.com/aiven/journalpump.

From Kafka, we export the logs to Elasticsearch for viewing and analysis. Some specific logs are also stored in S3 for long term storage for e.g. audit purposes.

Reply on news.ycombinator.com