The way it was described in the docs gave me the impression there is no acknowledgement of network writes - if that's true won't even clean shutdowns lose data sometimes?
Fuentd provides "buffer_type file" to buffer records on disk. Shutting down won't loose data. If you need to choose memory buffer for performance reasons, fluentd enables "flush_at_shutdown" option by default.
You would also want to use <secondary> feature. This lets you to write a buffer chunk to another storage if the primary destination is not available "retry_limit" times.
Those concerns would be solved by the document: http://docs.fluentd.org/articles/out_forward#buffered-output...
I am still interested in forwarder failure cases - I have replied to kiyoto's comment talking about the HA docs, which still talk about some other cases that can lose messages.
In this case:
* The process dies immediately after receiving the events, but before writing them into the buffer.
Is it possible to require acknowledgement that the log event has been written to the buffer? Is that separate to what require_ack_response does?
Where does it say this? I don't think this was ever the case for Fluentd.
>The way it was described in the docs gave me the impression there is no acknowledgement of network writes - if that's true won't even clean shutdowns lose data sometimes?
This is not true. All writes are acknowledged over TCP, at least between Fluentd and Fluentd.
It's in http://docs.fluentd.org/articles/high-availability#forwarder..., which says:
However, possible message loss scenarios do exist:
The process [log forwarder’s fluentd from the paragraph above] dies immediately after receiving the events, but before writing them into the buffer.
Is this document out of date?
No, in the described case, the message can get lost. This is a really unlikely scenario though. The only real-world case that I know of first-hand is using file buffer and somehow being unable to write to disk, possibly because the disk is full. Something like that can be prevented by a fairly routine set of server monitoring alerts.