Smarter log handling, the case for opportunistic logging

You could only keep the detail in memory for a particular window of time.

For example, keep a rolling window of two or three hours of trace logging in memory. If the number of errors in that window passes a threshold, write the full trace out to disk, otherwise just write the info logging.

You save on disk space while having detailed logging around errors.

The idea is to use an instance of logger to store messages related to some event, like serving a particular request or running a background job. If nothing happends, you discard everything and don't pollute logs. If you're not so lucky, it will flush all the messages, so you get a better picture of what was happening before the error. Compared to having just an info level logs that usually omit many relevant details.

No, you can prevent unbounded growth by using a pre-allocated cyclic buffer and just write into that.

We have prototypes of this in our software (but not production code yet) with buffers that default to 50MB. That gives a nice amount of debug info in the run up to any errors.

Whenever the buffer is dumped out you simply set the begin/end pointers of the cyclic buffer to be the same so that subsequent errors don't write out an entire copy of the buffer again, only the new log messages since the previous error.

What you are missing is not the storage considerations, but tying the application to I/O. Too often I set out to find the bottlenecks in a high load application and find out that the logging is the bottleneck yet again.