Clog: Making a homegrown ClickHouse log for $20 a month

I had a program which occasionally segfaulted (and even raised a SIGILL once, i forget how). By the time it segfaults, it's too late to get logging out (easily, at least). But i didn't want to write an ever-growing log of everything.

So, i did something a bit like the coroner. When the program started, it created a fresh log file, extended it to a certain size, and memory-mapped it. It then logged into this buffer, with new logging overwriting old (it wasn't actually a circular buffer; the program dropped a big blob of logging into the buffer at the top of its main loop).

While alive, the process never closed or msynced the mapping, and it was fixed size, so the kernel was under no particular pressure to write the contents to disk. But when the process crashed, the kernel would preserve the contents.

I admit i never benchmarked this, so i don't know whether it actually avoided excessive writes. But it seemed like a neat idea in principle!

It's been quite a few years, but when I started in telco, we had a vendor product that I think sort of worked like this. They were using a "micro-services" architecture within their nodes before it became popular. They also used a crash-only approach to writing software. So lots of asserts for unhandled / unexpected cases.

As I remember it, they wrote their crash handler to include the ring buffer of recent messages sent to the services. So whenever they'd get into an unexpected state, they'd just crash the process, and collect the ring buffer of recent messages along with the other normal things in a mini core. Made it so easy to track down those unexpected / corner cases in that platform.

This is a very common practice in embedded code, generally three things:

1. A ring of log-like objects (obviously not rendered strings, since that is a waste of CPU) that can be optionally included in a crash report in structured form that can be dissected later.

2. Compiler-generated enter/exit counters and corresponding table per module, modules linking themselves as init time to the master table, for performance counters [invocations or time spent]; dumpable on demand; lightweight and always on

3. a ring of logs - these actually being rendered logs plus indices into (1) - that have been otherwise rendered, so the retention cost is minimal and you can map back to log files otherwise provided.

The distinction between (1) and (3) should be obvious, but in case it is not, short circuiting log rendering for logs that should otherwise be dropped is a very important practice to avoid debug-level logs consuming the majority of CPU time.

Traditionally, all of these are trivially inspectable in a core dump, but usually you'd like a reduced crash report instead: less wear and tear on the flash and easier for users [and bug management systems] to juggle. Crash reports and cores obviously need to include an unambiguous version [typically a hash of the code rather than a manually managed version #, for dynamically linked ELFs, fingerprints of all libraries as well]; for cores you just make sure to compute this at start and keep it in memory reachable from a pointer out of main().

Modern CPUs do actually offer this for the most part, it’s called time travel debugging. Intel’s offering is called Intel Processor Trace. Although it’s not full input output logging.

Thank you for mentioning qryn! And yes, thanks to our growing community the rough edges should all be well rounded and we now support logs, metrics and telemetry formats with very decent results at a fraction of the complexity, without reinventing the wheel or adding new protocols and formats for people to adopt and learn. It's good if it just works right? And now with https://qryn.cloud we continue our mission towards a lightweight polyglot stack for core and edge observability - on top of the growing ClickHouse superpowers, as always!

Disclaimer: I work on qryn

Author here. The limit we hit with ES was the memory required to have anything approaching decent response times for queries. I've always tried to keep ES in memory but that becomes a pain, and honestly the defaults for CH just... work whereas ES ends up needing a hundred dials slightly tweaked to make it work (and keep working).

The new JSON type looks amazing though! It's probably time I update the code, but honestly it's just been silently chugging along in the background with zero maintenance for now.

I came here to say something similar. It has also occurred to me that vectordev hasn’t gotten much attention here on HN. It’s a great tool. You can listen on a TCP socket, write to Kafka in an upstream instance, then read from Kafka, transform data, and write to ES or whatever downstream. I’ve even recently completed tests where I could write from vector to vector at over 6 million records a second.

I don’t really mean to discourage the author of the post, I just realize this can be done with two simple vector configs.

In particular Vector can do simple transforms on the messages (or quite complex transforms using Lua script), and insert directly into Clickhouse.

I respect the simplicity of Redis here, but personally I would consider Kafka if I wanted to scale this setup further, maybe one of:

Vector -> Kafka -> Clickhouse (using Clickhouse's built-in Kafka support)

Vector -> Kafka -> Vector -> Clickhouse

This also lets you stream the logs live if you want to.

How good are they at opentelemetry support?

I like FluentBit, written in C and very light on resources.

Something the article leaves out is the volume of the logs ingested (GB) vs. just the number of log entries. He says "between 750K and 1M logs a day", but doesn't indicate how large those logs are.

Most commercial solutions are priced based on volume/size rather than qps or API calls.

If you use CH with s3 storage engine it will be considerably cheaper for same amount of logs

It depends on the solution, Loki for instance has support for it: https://grafana.com/blog/2019/08/13/lokis-path-to-ga-live-ta...

With that said unless you tune your query it's very easy to get overwhelmed once you scale a bit and start getting larger log volumes.

Not really, but then this isn't used for single service apps. It'd be difficult to tail anyway when your app grows beyond a few services.

Clickhouse has experimental feature enabling a Watch https://clickhouse.com/docs/en/sql-reference/statements/watc...

Yes, but YMMV...

https://cloud.google.com/logging/docs/view/streaming-live-ta...

My biggest worry with such a solution is a program going crazy and blowing up my bill at the end of the month.

Less a concern for enterprise projects where you can setup cost monitoring, but terrifying for hobby projects running on credits. I can't find the article, but I remember reading something similar happening to someone using GCP not very long ago.

Is there a front end to Cloudwatch that can visualize the results in a way so that we don't need to give AWS account access to the viewer?