Hacker News Reader

Comment by cbsmith | Hacker News Reader

Except you can scale up and beat that performance significantly if you just the standard syslog wire protocol to a central log server. That avoids a lot of independent append operations that you are counting on staying in the OS buffer despite whatever other IO might be going on, and you're also burning a lot of CPU (and space) transforming things to JSON using an inefficient JRuby runtime.

We make this stuff harder by rebuilding functionality on top of systems that already have the functionality.

View on HN

RELP is very much standard syslog wire protocol(which is just plain tcp anyway), but with some extra stuff to avoid losing data. Just fixing tcp issue here, where data could be ACK-ed into OS receive buffer and the app could be killed before offloading into a persisten storage. Not losing data is very important in log shipping.

The json that syslog wraps data in is actually using the same syslog protocol - nothing is chaging here. It's encapsulated within the same protocol. The only reason to unwrap json is when you need to start structuring your data for BI views, your milage may vary. The key is that these two are de-coupled.

Because you work on a basic level networking / storage layers its very easy to reason and build reliable pipelines according to your needs. As you say — you can't beat that.

PS: The pain starts when you need to deal with multiline logs like java stack traces but then you just work with devs to log everything into json.

Edit: split comment, more info

That works fine until your centralized server falls over, either because you're scaling to hundreds or thousands of machines and your logs are chatty or someone goofs and starts logging kilobytes of data on every request.

If you care about that, it's very easy to have redundancy there. In fact, the target pipe you route to on each container/system/etc. can be fully virtualized and routed as needed by... syslog. ;-) If you want to go crazy you can wrap it in rate limiting which is done for you by... syslog.

There would be usually multiple syslog sink servers so there is never a single point of failure. This can be configured in various ways either dns round robin, load balancers or even multiple destinations configured on the client side, rsyslog can iterate over a list in case of failures.

If there is log storm coming from misconfigured app depending on your traffic levels the sink servers should give you a lot of space for an on-call engineer to be notified that something is off and adjust / rate limit / drop accordingly.

Edit: udated with more info

Reply on news.ycombinator.com