Hacker News Reader

Comment by anygivnthursday | Hacker News Reader

I am not sure if this is what they mean, but e.g. with Micrometer in Java you can instrument your code once with observations that produces observation events, then you can register handlers that can turn them into metrics, or logs, or traces without having to instrument your code three times.

https://docs.micrometer.io/micrometer/reference/observation....

View on HN

The problem is not the instrumentation but the way everyone of them work.

A metric is a point in time. A metric is very small but you have a lot of them.

A log is when something is happening but you need to log it out. A logline is heavy and has a lot of context. User id, message, etc.

A trace needs to start at the request level and tracing until the response. This is the slowest and heaviest operation.

How do you decide when to suddenly do the trace and send it? IF you always do the trace, you have to pay for the overhead of that tracing constantly.

Logs and metrics are both derived from events. A log takes the whole event and records it somewhere. A metric takes some numeric value from the event, aggregates it over time, and records it periodically. You can reconstruct a metric from logs for the underlying events.

A trace is a period of execution between two events. You could record a trace as a pair of log entries, or one log entry at the end. You can then reconstruct a trace from those log entries. If you want to associate multiple spans, and separate log entries, within a trace, you use a shared ID, which is just the same as a context entry for logging.

All three of these pillars are just ways of looking at events. They are not fundamentally different at all. This is a mistaken idea in "Observability 1.0" whose correction is the basis of "Observability 2.0".

The pillars still have their uses, but the choice between them is really a non-functional one - storing a log entry for every event might be too expensive, so just store metrics instead, and index every log entry so it can be correlated with nearby ones might be too expensive, so just store specific traces instead.

This is the literally the "everything is a graph" argument from database architecture. The conceptual abstraction fails badly because it has to be implemented on real silicon that imposes constraints not considered in the abstraction.

Logs, metrics, and traces are all derived from raw events but none of them are intrinsically discrete events in a systems engineering sense. They are all different data models with different patterns of traversal over raw events. As data model, you need to build secondary indexes over the raw metrics to reflect the orthogonal data access patterns depending on if you are evaluating them as logs, metrics, or traces. This famously has poor scalability and performance.

In analytical processing we largely manage the inherent performance and scalability issues using denormalization, which allows processing pipelines with very different requirements to be optimized independently. Or in this context, treating logs, metrics, and traces as unrelated things with independent infrastructure.

"Observability 2.0" deeply embeds an architectural assumption that all systems are small. It is not a tractable architecture in high-scale or high-performance systems.

Real silicon has a long history of destroying beautiful conceptual abstractions in software engineering.

You are conflating the challenges of ingesting and querying at large scale with the what the original comment is about, which is emitting them more easily.

I don't see them as separate issues. Emitting them directly runs into the inherently poor memory locality (and potentially concurrency) of trying to produce logs, metrics, and traces from the same underlying event data representation.

It is only "easy" if performance and scalability don't matter.

> Logs and metrics are both derived from events. A log takes the whole event and records it somewhere. A metric takes some numeric value from the event, aggregates it over time, and records it periodically. You can reconstruct a metric from logs for the underlying events.

No, metric is just value. Some are derived from events (like histogram/rate of given event duration) but others are wholly independent (like returning app's CPU/memory usage)

The app's memory usage is an aggregation of the alloc/free events. I think the original point was that all of the metrics, traces and logs are conceptually the same but for efficiency, we store less data in each place, not the full history. Personally, for the systems I work on, having an easy way to turn logs into metrics and vice versa, without deciding up front, would be a slight benefit.

A clock ticking every second is generating an event every second.

If you sample the CPU usage at 1Hz, the metric is attached to the tick event.

A metric is not event based.

You don't have a metric 'person logged in' because you would need to scrape the metric at the moment a person logged in.

You have a metric called 'overall people have logged in so far' and you do math on it.

The 'person logged in' is an event you log out.

Technically, you can use the same places in the code where you stop/start/fork traces to also be the places where you increment the counters/gauges, etc. Which I think the GP was alluding to when describing the micrometer solution. Similarly, you can derive metrics for log lines without having to emit the actual log lines.

Then separately you can have log levels or verbosity levels that control to which level you actually emit traces/logs and/or roll up metrics.

At that point you almost might as well just log everything. The decision logic is likely about as complex as just doing it. Then I suppose you have a watchdog task that fires off every, say, 15 minutes or an hour or something, looks at the collected data, and either decides to keep it or trash it while recording a tiny "nothing interesting" datapoint.

Loghandling is quite resource intensive.

All the log ingestion systems i have seen were bigger elastic search clusters.

What? All of this has been solved for a long time. How do you think hyperscalers do this?

Search keyword: "Adaptive sampling"

Adaptive sampling is not tracing, its sampling.

Tracing traces a particular event.

I'm quite aware of the difference between sampling, tracing and profiling.

No... Adaptive sampling is a family of statistical methods to choose an appropriate decimation strategy for arbitrary events based on real-world occurrence distributions.

It can be applied to tracing, metrics, logging ("sampling") and profiling.

Reply on news.ycombinator.com