Sorry but then your monitoring system is broken? Why is its operation tied to log chunking?

Multiple logfiles for a day are harder to search

Not really. Often enough interesting events span multiple days (or cross midnight) anyways. Thus regardless of chunking you have to be prepared for that case.

What benefits does a split logfile provide?

I'll give you that "one file per day" seems easier at first, but I've been bitten by that too often. A script to identify the file-range for a given time-range via binary search is really easy to write, has only to be written once, and the benefits of fixed chunks are:

Easier handling. If you need to offload "50GB" from a given log-host then that's easier to do when you know that will be 50 files.

You don't risk a scratch disk running full (i.e. those small and expensive spindles that take the initial load) when the application decides to have a bad-hair day and put out 10x the normal volume for a while (e.g. interleaved with stack traces).

>>fetching an arbitrary header from the right file for a given context can be rather painful. That's been a solved problem for a very long time. Just as recombining a separate trace log with a main app log based on time stamp is a solved problem.

I said painful, not impossible. You say you're dealing with 5GB/hour logs, then you must be well aware how long even the simplest merge-job takes on the slice of a single day?

>> When a serious error occurs then as much context as feasible must be provided Yes this is the extraordinarily rare case where stack traces can be useful.

We must be living in different worlds then.

When errors pop up in the main logfile, which happens frequently, then the last thing I want is to stitch together context from other files.

A mere grep on a day's worth of logs takes about 10 minutes for us. Any kind of text-processing or "for each line perform lookup in $otherplace" quickly pushes that into the hours.