NGINX syslog-ing without breaking the bank or patching the code
syshero.org
syshero.org
" CloudFlare generates 50gb/s of logs globally and have handled collecting this volume in two ways. Historically the logs are sent to a local syslog-ng through the use of a PIPE and the forwarded to central logger. This can be done with nginx with no patches by just treating the PIPE as file. Just make sure you do a little buffering inside nginx.
access_log /dev/nginx_access log_format_name buffer=64k flush=10s;
Since this is a pipe there is still some blocking IO, but no worse off then writing to local file."
I would still configure nginx to write to a file and have the consumer tail it to avoid this situation. Properly-configured log rotation can keep the file size within reasonable bounds.
Do you just crontab logrotate for short intervals, and will that trigger "daily" rotations each time for other configurations?
But how likely is that to happen in any circumstance that doesn't also bring down nginx anyways? Obviously in the case of a large setup like that they don't care, it just gets removed from the pool regardless of why it is failing.
If the other side of the pipe closes (e.g. syslog-ng), your process will hang once the I/O buffer fills up, which is generally 4KB.
You especially need to keep this in mind, when syslog-ng restarts (pipe closed, pipe reopened). You'll probably glance quickly to see that syslog-ng is running, and the pipe is present, but wonder why nginx is not serving traffic.
Or worse, be fooled into thinking nginx is fine because it was serving traffic up until it served 50 requests (or however many reqs it takes to generate 4,000 bytes of access logs), then stopped.
Who watches the watchers? What happened to writing things that don't break?
The major difference between a thing that might go wrong
and a thing that cannot possibly go wrong is that when a
thing that cannot possibly go wrong goes wrong it usually
turns out to be impossible to get at or repair.
Referring to a human as a "devop" is equivalent to calling someone an "agile".
Realistically its just a job title that means paying technical architects less money...
We write the logs to disk and then use rsyslogs imfile feature to read from there. Your approach has the advantage of not requiring disk writes.
BTW, we're not in Toronto, but we're hiring and happy to accept remote workers from that timezone :)
oh and don't remember to rotate them!
Your solution works for centralising the log, but it doesn't solve the local i/o issue as the file is still being written (which was the problem of the original article).