Hacker News Reader

Graphing metrics and doing transformations/comparisons is (much) faster. Yes, metrics do have to be defined first but in my experience that's a non-issue since the things you want to monitor are usually immediately obvious during development (e.g. request-response times, errors returned, new customers acquired, pages loaded, etc.).

With that being said, it's not a mutually exclusive situation. You can have both. However, some logs used for plotting metrics have near-zero debugging value (e.g. a log line that just includes the timestamp and the message "event x occurred"). Those kinds of logs should be fully converted over to metrics.

Some other logs however are genuinely useful (e.g. an exception occurred, the error count should be incremented, and this is the stack trace).

I don't understand why "some logs used for plotting metrics have near-zero debugging value (e.g. a log line that just includes the timestamp and the message "event x occurred"). Those kinds of logs should be fully converted over to metrics."

What is the difference between logs and metrics then?

Logs are pieces of text and metrics are well defined data points?

I can represent both using the same log format. I can also extract the logs with well defined structure into a columns store or inverted index for fast querying and plotting.

The entire distinction of logs and metrics and keeping one vs the other reeks of strong premature optimization by the software community. Storage is cheap, just dump the raw logs to s3 and run etl on them to extract meaningful metrics.

Logs, metrics and traces have the same representation - text or some kind of json yo have it structured. Metrics are just logs with well defined shema. Traces are logs with correlation ids in them to allow for joining between logs coming from different services.

It's just a data problem, nothing else. However, people keep overpaying for complicated services for observability that they don't need. When they split it into logs, metrics and traces they have to connect their programs using proprietary connectors to these external services, adding vendor lockin and potential for failure when the observability service has downtime. Instead of just dumping logs into stderr as json objects as intended by the unix philosophy.

As you correctly point out, everything you can do with metrics can be achieved via logs if you have enough compute and I/O. But that ignores the reality of what happens when you do indeed have too many logs to use something fancier like column store/inverted indices as you point out? I agree that in the vast majority of cases, it's likely fine to just take the approach of using logs and counting. But plenty of of developers (particularly here on HN) are in that overall small slice of the overall community that does have a genuine need for greater performance than is afforded by some form of optimized logging.

Likewise, traces are indeed (as you point out) functionally just correlating logs from multiple services which is akin to syntactic sugar. But again, that's precisely its value _at scale_: easing the burden of use. I've personally seen traces that reach across tens of services in an interconnected and cyclical graph of dependencies that would be hellish to query the logs for by hand.

Splunk being “too expensive” has nothing to do with finances.

From my second FTE position I’ve been dealing with the consequences of people logging things that were important to them a year ago that they no longer care about but can’t be bothered to remove either due to hoarding dynamics, project scheduling dynamics, or effort/reward dynamics (“that’s an old feature/bug, and I get paid to work on new things”).

Stats get aggregated with their own kind, while logs get jumbled together with everything else. The median value of old log messages is negative, while that of old stats is closer to zero.

If you put a bunch of data into a log message, its initial value is higher but so is the negative slope of the line. If you don’t put a bunch of data into the log message, what’s the difference between logging it or recording a stat?

Part of the unspoken assumption of logs is that I can read cause and effect chronologically in the logs. That ceases being true the moment you have a distributed systems. Getting reliable sub-second clock skew is expensive. Getting it down below speed of light delays within a single rack is pure fiction.

Anyone implementing stats on a single server application has bigger problems than logs vs stats, so distributed is practically a pre-req for even having a stats system.

Log messages also have to be defined first.

I find it much harder to guess what log messages I will need in the future compared to what metrics.

Metrics are easy: for any resource, service, component, and queue (explicit or implicit), you want to know cumulative arrivals, departures, timeouts, errors.

That takes you most of the way there for most things.

You can add counters for significant events too, but they tend to be automatically included by the above. (Basically departures from the component that trigger the event.)

> The solution goes - Splunk is too expensive, let's ditch logs altogether and hope we capture some metrics and plot them in nice dashboards. Instead of dumping the logs highly compressed into cheap s3 and running some Snowflake or Spark on it later.

I don't usually promote on HN but this is exactly why we built https://axiom.co! We've been working on this problem for some time, essentially allowing schema-less/index-free ingest, S3-based storage in a highly-efficient format, and then querying with a Splunk-like (specifically Kusto-inspired) language via serverless functions.

We built it because we also realised we would avoid or think too much about logging (cost, scaling, retention, etc) which led to compromises either in our monitoring or later when we wanted to dive in and try and draw some insights/analytics from that kind of data.

> but I wonder why move away from the concept of metrics extracted from logs to discarding logs and storing just the metrics?

This is a _very_ good question.

Getting from log to meaningful dashboard full of metrics is very hard to do well. Its also very hard for a normal person(ie non programmer) to ask the right question of a log stream.

For example if you give me a log stream of a web server, and want me to raise an alert when an instance's health check takes longer than n milliseconds, it would require regex to get the host name, regex to get the health check, and more regex to get the response time. All of those steps are hard and require testing. Moreover its fragile, any kind of format change is really easy to slip in and cause your monitoring to break.

So your next question might be, why replace splunk with a metric based solution?

Thats the thing, you don't! logs are really really useful for what specifically went wrong. But they are not very good at telling you what is _currently_ going on over n number of services.

when implementing a service/program/things, I ask the developers to think about the information that is useful to log. Be that response time, number of peers connected, cache size, etc etc. Then instead of just dumping them to logs, you'd push them to a metric library and get that to look after it.

this means that you are consciously thinking about an number that best sums up the thing you care about. This then frees up logging to explain _why_ something has happened, rather than being rigidly defined because you'll break a dashboard.

Think of is as this:

dashboard: shows you if a service is happy, which is an aggregate of many smaller programmes.

Something goes wrong, response time of a microservice is above limits, you then track down when exactly the metric went wrong, and fire up splunk to look at the logs.