It's useful if you want to log to a centralized logging server. This helps to have all your logs in one place and also keeps the logs safe, if someone breaks into your server.
Why not use something like multilog or svlogd and wire up a tiny processor for it to kick logging data over someplace using something like rsync?
To boot, syslog is annoying to tune, depending on your particular implementation. rsyslog has a default buffer limit of 2k, whereas other syslog implementations (IIRC, syslog-ng and Solaris syslog at the very least) have default buffer limits of 1k, and this might not be obvious until you're running up against that and make the shocking discovery that you're losing data.
On an nginx server that services 2TB/mo worth of transit (which is distinctly possible since I've got infrastructure in production that does this), there's a good chance that you'll be stretching some of these limits a bit.
rsyslog and syslog-ng have support for TCP
> Why not use something like multilog or svlogd
Additional point of failure.
> and wire up a tiny processor for it to kick logging data over someplace using something like rsync?
Additional point of failure (processor); additional point of failure (rsync/ssh); non-realtime log replication (which is bad for breaking/progressive system failure/etc).
> To boot, syslog is annoying to tune, depending on your particular implementation.
All of the examples you list are easier to learn about, tweak, and monitor than the suggestions you've proposed, however.
> On an nginx server that services 2TB/mo worth of transit (which is distinctly possible since I've got infrastructure in production that does this), there's a good chance that you'll be stretching some of these limits a bit.
If you're dealing with 2 TB/mo in transit, you're probably capable enough to understand the risks with centralized log management and mitigate/monitor them ahead of time.
On Linux, sending UDP to localhost is very reliable and fast, essentially going through kernel buffers with very little overhead. You will only see dropped data if the system is extremely overloaded. I did some testing, a few years back, and was not able to induce packet loss on localhost.
The usual way to set up centralized logging with syslog is to have each node run a local syslog daemon (eg., RSyslog), which then buffers the data and streams it to a central syslog daemon using a more reliable protocol such as RELP [1] over TCP.
On the other hand if you use stream sockets, the producer will either block or be told that the consumer is not ready to read any more data (beauty of TCP). In either case, TCP produces enough overhead compared to UDP to slow down the actual useful part of your application, which is often not desirable.
Neither one of these is a good solution as either your consumer or your producer needs to keep their own very large buffers to accommodate spikes in traffic. Ideally, you do this anyways to ensure that you hold onto all the packets you received.
Having said that, I don't know exactly what rsyslog does so I cannot say if this would actually be a problem for it.