Thanks for the answer.
Just for clarification,
1. For my case (yea, they are kind of required audit data), I need the log to be able to not loss when sending to logstash or kafka, probably this should be handled in client side by buffering the data before sending to logstash?
2. For volume "at scale", my point is just for clarify whether this logstash could be good to be used for production for big volumes. Maybe can be like 100GB per day
What kind and required by whom? Most of such things are "if you have all, good, if half an hour is missing, we'll do without those", not something akin to transaction data in relational database where you cannot lose even a single byte.
> I need the log to be able to not loss when sending to logstash or kafka
What is "not to lose logs"? Is losing a single record OK? Is losing 1% of logs OK? How long network problems should be tolerated? What to do if such network problems are longer than anticipated and disk buffer is running out of space?
You have said virtually nothing about what your logs consist of and what kind of reliability you require from them.
That's great question
> What kind and required by whom? Most of such things are "if you have all, good, if half an hour is missing, we'll do without those", not something akin to transaction data in relational database where you cannot lose even a single byte.
So we will do log analysis from the data (counting, sum of the amount from the data, etc). So ideally need 100% precision and we will do the analysis in daily basis
> What is "not to lose logs"? Is losing a single record OK? Is losing 1% of logs OK? How long network problems should be tolerated? What to do if such network problems are longer than anticipated and disk buffer is running out of space?
In my case, losing single record is significant enough because we will do calculation for those logs. For the network tolerance, I think that's very good question, I'll take that as my consideration too. But the network problems that I mean before is the intermittent one like < 1 minutes and with assumption the data buffered is not until making the server running out of space