Lossless Log Aggregation – Reduce Log Volume by 99% Without Dropping Data
bit.kevinslin.com
bit.kevinslin.com
Log message template parameters would have to be fully represented in the message, it would be way too much work to register the arbitrary values that get thrown in there.
Next logical step is to serialize to something more efficient than JSON— should be dead simple, it’s a template followed by N values to sub into the template, could be a proprietary format, or just use something like protobuf.
It’s better than compression, because the data that was being compressed (full text of log message template) is just not even present. You could still see gains from compressing the entire message, in case it has text template values that would benefit from it.
I get it, we lost human readability, which may be too big of a compromise for some, but we accomplish the main goal of “make logs smaller” without losing data (individually timestamped events). Besides, this could be made up for with a really nice log viewer client.
I’m sure this all exists already to some degree and I just look dumb, but look dumb I will.
In fact, I'm going to build a toy one in python!
I suggest building it as a normal python logging handler instead of totally custom, that way you don't need a "text" toggle and it can be used without changing any existing standard python logging code. Only requires one tweak to the idea: Rather than a template table at the start of the file, have two types of log entries and add each template the first time it's used.
Drawback is having to parse the whole file to find all the templates, but you could also do something like putting the templates into a separate file to avoid that...
I am writing code on my machine, running one query at a time. I can easily view the logs and spend a lot of time looking at them.
I am running test suites, running thousands of queries. It's harder, but I will still view the logs around failures.
Then I am taking the very code, and pushing it to prod. Should my logs be suddenly completely different in this case?
(The right answer of course is to have "log-to-text formatter", either running in-process or as a separate post-processing step. But it better produce nice-looking, human-readable logs, or every format will be "Message: %s" )
The problem with weird log formats is recreating all the neat stuff you can do with tooling not necessarily just being able to open a file in a text editor.
The benefits to this approach is it's dramatically easier to index and cheaper to store at any scale.
One thing I think people don't appreciate about logging efficiency is it enables you to log and store more and I think many don't appreciate how much even modest amounts of text logs can bog down systems. You can't read anything, but you filters easy and powerful and you can't filter something that doesn't exist.
How did your solution work out in terms of adoption by others? Was it a large team using it? What did those people say? Really curious!
To answer your question: the prototype was never finished, but the concepts were adapted to a production version and is used for structured events in a semi-embedded system at my work.
CSV without fixed columns would be fine for that.
> require that log message templates be registered in a schema, and then a byte or two can replace the text part of the message.
Pre-registering is annoying to handle, and compression already de-duplicates these very well. Alternatively the logger can track every template logged in this file so far, and assign it an integer on the fly.
Logging is unstructured and free-form by design. The observability events that can be expressed with a fixed format are metrics events, which are already serialized in a trivial format. See line formats from services like Statsd.
There were a bunch of problems/irritants mostly stemming from the fact that the format become stateful. Every log needed to have a schema (or repository) available. Abbreviations and date offsets meant that the log contained meta information ... for example, assignment of a compact abbreviation to a string in anticipation of using that abbreviation from that point on. This meant that the log could not be arbitrarily lopped off.
And to my chagrin, I found that simply gzipping a json stream made it almost as compact! That's when I figured it wasn't worth it. I'd probably have investigated more if there was CPU or memory bandwidth pressure in that situation (due to creating more data just to compress it).
Genuinely lossless compression algorithms like gzip work pretty well.
I worked at a magnificent 7 company that compressed a lot of logs; we found that zstd actually did the best all-around job back in 2021 after a lot of testing.
If you used something like sequential IDs (even in some UUID format) it can compress pretty well.
I laugh every time I see it.
If I want something like per-transaction rollup of events into one log message, I build it and use it explicitly.
Regarding dropped timestamps: if log order is preserved within an aggregation, that would be sufficient for me. I'm less concerned with millisecond precision and more interested in sequence. I'm making a big assumption that aggregation would be limited to relatively short time horizons on the order of seconds, not minutes.
Grouping related logs sounds like a big win to me in general. Certainly I can perform the same kind of filtering manually in a log viewer, but having logs grouped by common properties by default sounds like it would make logs easier to scan.
I've never been involved in billing for logging services so I can't comment on the efficiency gains of log aggregation vs zstd or gzip as proposed by some other commenters.
If you're going to agregate your logs, you're much better off converting them to metrics _on device_. it makes comparison much easier, and storage and pensioning trivial.
Metrics tell you when and what went wrong, but unless you've got lots of coverage, its less likely to tell you _why_
This works for when you cannot change the log source too (e.g. third party component or even legacy hardware that may be syslog only).
If you process logs in a smart way you have to have assumptions about them. When the assumptions are broken the data might get missing (or you can lose the correlation between logs that are close together).
That's why I prefer raw unprocessed logs.
For example let's imagine you have a bug with stale transaction id in the first example. You probably won't notice when it's grouped by transactionId. If it's raw logs you might notice the order of the logs is wrong.
Or maybe the grouping key is missing and your throw NPE and crash the app or skip that line of logs. I've seen both happen when I got too fancy with logging (we had async logger that logged by inserting into a db).
The other cases are more interesting, and pre-aggregation of all logs related to a correlation ID can be really helpful when debugging a specific incident, but it does seem like this proposal is the same basic trade-off around size and performance as with virtually any form of compression.
So in theory we could uniquely identify all logs as a much more compact binary representation + a lookup table we ship with the executable.
Sure, zipping your logs gives you a LOT, and you should do it, but if the result disappoints it is not the end of the road, at all.
IP addresses are obviously part of the formatting data, not the fixed strings.
IPv6 is used internally a lot more than externally, so I would expect to see a LOT of commonality in the network ID fraction of the addresses- essentially all the bits of representing your IPv6 network ID get reduced to the number of bits in a compression token, in the worst case. In the moderate case, you get a few chatty machines (DNS servers and the like) where the whole address is converted to a single compression token. In the best case, you get that AND a lot of repetition in the rest of the message, and you reduce most of each message to a single compression token.
It's hard to explain if you haven't actually experimented with it, but modern variants of LZ compression are miraculous. It's like compilers- your intuition tells you hand tuned assembly is better, but compilers know crazy tricks and that intuition is almost always wrong. Same with compressors- they don't look at data the same way you do, and they work way better than your intuition thinks they would.
It was hard to figure out how to use it without documentation so it wasn’t very popular. No idea if they still ship it in the DDK.
It was a preprocessor that converted logging macros into string table references so there was no runtime formatting. You decoded the binary logs with another tool after the fact.
Vaguely remember some open source cross platform tool that did something similar but the name escapes me now.
Is be curious how that stacks up against something like zstd with a predefined dictionary.
Originally there was only one system-wide application event log and you needed to be admin to install your message definitions but it all changed in Vista (IIRC). I'd lost interest by then so I don't know how it works now. I do know that the event log viewer is orders of magnitude slower than it was before the refit.
ETW is for high speed general purpose logging of low-level events with multiple collection mechanisms including realtime capture.
This is a logging library that does lazy formatting, - https://defmt.ferrous-systems.com/
In most modern distributed tracing, "observability", or similar systems the write amplification is typically 100:1 because of these overheads.
For example, in Azure, every log entry includes a bunch of highly repetitive fields in full, such as the resource ID, "Azure" as the source system, the log entry Type, the source system, tenant, etc...
A single "line" is typically over a kilobyte, but often the interesting part is maybe 4 to 20 bytes of actual payload data. Sending this involves HTTP overheads as well such as the headers, authentication, etc...
Most vendors in this space charge by the gigabyte, so as you can imagine they have zero incentive to improve on this.
Even for efficient binary logs such as the Windows performance counters, I noticed that second-to-second they're very highly redundant.
I once experimented with a metric monitor that could collect 10,000-15,000 metrics per server per second and use only about 100MB of storage per host... per year.
The trick was to simply binary-diff the collected metrics with some light "alignment" so that groups of related metrics would be at the same offsets. Almost all numbers become zero, and compress very well.
HTTP/2 would also improve efficiency because of its built-in header compression feature, but again, I've not seen this used much.
The ideal would be to have some sort of "session" cookie associated with a bag of constants, slowly changing values, and the schema for the source tables. Send this once a day or so, and then send only the cookie followed by columnar data compressed with RLE and then zstd. Ideally in a format where the server doesn't have to apply any processing to store the data apart from some light verification and appending onto existing blobs. I.e.: make the whole thing compatible with Parquet, Avro, or something other than just sending uncompressed JSON like a savage.
Weird perspective, yours.
It would cut costs a lot if the source agents did this (pre)processing locally before sending it down the wire.
The answer to "this thing is horrendously inefficient because of misaligned incentives" isn't to be frugal with the thing, but to make it efficient, ideally by aligning incentives.
Open source monitoring software will eventually blow the proprietary products out of the water because when you're running something yourself, the cost per gigabyte is now just your own cost and not a profit centre line item for someone else.
You can see a fairly general explanation of the concept here: https://messagetemplates.org/
But someone who has tried to wrangle gazillion row dumps from a variety of old msgpack protobuf etc and make sense of it all will hate it.
Zipped text formats are infinitely easier to own long term and import into future fancy tools and databases for analysis.
Of course, performance etc are a separate story but as far as the shape of the data goes, that's what the solution is
A log is universal, and can be projected to relational, document etc. Which is why most relational databases are built on top of them.
I would imagine the size reduction would be much better than the 40% Kevin showed with his approach.
As for volume, I don't know if volume is a problem. With logs I either need to look at one log entry to get it's details, a small handful to see a pattern, or aggregate statistics.
I can do all of those things wether the volume is 1x, 100x, or 100000x.
In other words, size matters to me, but I don't care about volume.
On the other hand, for cases when we use tools that charge for uncompressed size or message count, then Kevin's approach could be a useful cost-saving measure.
IME, I've found sampling simpler to reason about, and with the sampling rate part of the message, deriving metrics from logs works pretty well.
The example in the article is a little contrived. Healthchecks often originate from multiple hosts and/or logs contain the remote address+port, leading to each log message being effectively unique. So sure, one could parse the remote address into remote_address=192.168.12.23 remote_port=64780 and then decide to drop the port in the aggregation, but is it worth the squeeze?
But, in general, I think we agree -- all good!
That's reasonable in a reasonable environment. Alas, I worked in large legacy enterprises (like banks etc) where storage space is at much more of a premium for reasons.
You are right that sampling naively works better for metrics.
For logs you can still sample, but in a saner way: so instead of dropping each log line with an independent probability, you'll want correlation. Eg for each log file for each hour only flip one weighted coin to decide whether you want to keep the whole thing.
That same key property is not true for logs. Logs cannot be aggregated without loss of information. By definition. This isn't up for debate. You can collect logs into groups, based on similar properties, and you can decide that some of those groups are more or less important, based on some set of heuristics or decisions you can make in your system or platform or whatever. And you can decide that some of those groups can be dropped (sampled) according to some set of rules defined somewhere. But any and all of those decisions result in loss of information. You can say that that lost information isn't important or relevant, according to your own guidelines, and that's fine, but you're still losing information.
A slightly more complicated rules is: drop your logs with increasing probability as they age.
With any rule, you will lose information. Yes. Obviously. Sampling metrics also loses information.
The question is what's the best (or a good enough) trade-off for your use case.
Sure.
> A very simple rule is to just drop everything older than a threshold.
Yep, makes sense.
> (Or another simple rule: you keep x GiB of information, and drop the oldest logs until you fit into that size limit.)
Same thing, it seems like.
> A slightly more complicated rules is: drop your logs with increasing probability as they age.
This doesn't really make sense to me. Logs aren't probabilistic. If something happened on 2024-11-01T12:01:01Z, and I have logs for that timestamp, then I should be able to see that thing which happened at that time.
> Sampling metrics also loses information.
I mean, literally not, right? Metrics are explicitly the pillar of observability which can aggregate over time without loss of information. You never need to sample metric data. You just aggregate at whatever layers exist in your system. You can roll-up metric data from whatever input granularity D1 to whatever output granularity e.g. 10·D1, and that "loses" information in some sense, I guess, but the information isn't really lost, it's just made more coarse, or less specific. It's not in any way the same as sampling of e.g. log data, which literally deletes information. Right?