Hacker News Reader

> what do you do when you can not record everything you want to

The thesis of the article is that you should use events as the source of truth, and derive logs, metrics, and traces from them. I see nothing logically or spiritually wrong with that fundamental approach and it's been Honeycomb's approach since Day 1.

I do feel like Charity is ignoring the elephant in the room of transit and storage cost: "Logs are infinitely more powerful, useful and cost-effective than metrics." Weeeelllllll....

Honeycomb has been around a while. If transit and storage were free, using Honeycomb and similar solutions would be a no-brainer. Storage is cheap, but probably not as cheap as it ought to be in the cloud.[1] And certain kinds of transit are still pretty pricey in the cloud. Even if you get transit for free by keeping it local, using your primary network interface for shipping events reduces the amount of bandwidth remaining for the primary purpose of doing real work (i.e., handling requests).

Plus, I think people are aware--even if they don't specifically say so--that data that is processed, stored, and never used again is waste. Since we can't have perfect prior knowledge whether some data will be valuable later, the logical thing course of action is to retain everything. But since doing so has a cost, people will naturally tend towards trying to capture as little as they can get away with yet still be able to do their job.

[1] I work for AWS, but any opinions stated herein are mine and not necessarily those of my colleagues or the company I work for.

> The thesis of the article is that you should use events as the source of truth, and derive logs, metrics, and traces from them.

That was the main point? Man, the article goes on and on about definitions and semantic versioning and how we should call observability "olly", but glad to hear there's actually a coherent thought behind it.

Also, I don't think the "it's too expensive to log everything" argument should be true. Storage is insanely cheap and you can fit a whole lot of information inside a single MB (even more so when using compression).

Of course everything is events. The distinction is how do and should they degrade when you can not record everything (and secondarily how we should interpret and correlate them, though that is more in the structured/schema aspect).

I personally believe you should deploy systems with robust, comprehensive, automatic instrumentation that you analyze offline, which appears to match what the author is advocating for (except maybe the automatic part). But, that is orthogonal to the author’s claim that it is some sort of groundbreaking revolution in observability. It is not, it is just a scheme that assumes you only need to tread the happy path.

However, if you are able to stay on the happy path, then you should absolutely prefer a scheme in the same general vicinity as what the article proposes. Personally, the things the author links to leads me to question the efficiency of the implementations (compute, bandwidth, storage bottlenecks become worse with inefficient implementations) which makes it harder to stay on the happy path, but the core idea is very effective on the happy path.

Yes, I'm quite sure the CTO of a leading observability platform is simply confused about terminology.

It is not impossible that this is the case (at least in GP's view). Companies in the space argue between logs and events as structured or unstructured data and how much to exploit that structure. Unstructured is the simple way, and appears to be the approach that TFA prefers, while deep exploitation of structured event collection actually appears to be better for many technical reasons but is more complex.

From what I can tell, Honeycomb is staffed up with operators (SRE types). The GP is thinking about logs, traces, and metrics like a mathematician, and I am not sure that anyone at Honeycomb actually thinks that way.

I think the Veserv was too nice in their comments - the CTO is not "simply confused", she is specifically trying to introduce confusion to separate Honeycomb from competitors.

See, under common definition of observability, Honeycomb is just another platform with upsides and downsides. But if you believe the original post, Honeycomb is "2.0" while everything else is "1.0". So why are you even bothering with feature comparison for with other platforms? Are you _really_ going to go with observability 1.0 (a.k.a. most of Honeycomb's competitors) when 2.0 is out? Just sign up already, preferably for most expensive plan so you can record all those events you'll be generating.

jesus christ ok

To be fair, the first heading and section of TFA says no one knows what the terminology means anymore.

Which, to GP’s point, is kind of BS.

When in doubt, call it a data lake.

Sounds to me like that's just a database but with extra steps

Database, with schema left as an exercise for the reader

> Logs are independent. When you can not store every event, you can drop them randomly. You lose a perfect view of every logged event, but you still retain a statistical view. As we have already assumed you can not log everything, this is the best you can do anyways.

Usually, every log message occurs as part of a process (no matter how short lived), and so I'm not sure it's ever the case that a given log is truly independent from others. "Sampling log events" is a smell that indicates that you will have incomplete views on any given txn/process.

What I find works better is to always correlate logs to traces, and when just drop the majority of non-error traces (keep 100% of actionable error traces).

> When you can not store every event, you want to drop randomly at the whole trace level.

Yes, this.

Yes, you can probably combine traces and logs into a single category, and really what are traces but a collection of related logs with information about the relationship between them.

But metrics are an entirely different beast.

Sure some metrics can be derived from "events". But what about thinks like CPU and RAM usage? Do you want to record an event for every packet so you can measure network performance? And as mentioned above, you generally don't want to sample the things you have metrics if.

If I were being pedantic, I'd say that the metric of "CPU usage" comes from the event "measuring the CPU usage" :p

> you generally don't want to sample the things you have metrics [o]f.

Metrics are a sampling of a counter/gauge.

I wonder if anyone has utilized trace capabilities like head and tail based sampling _for logging_. You could potentially have the same logic used to determine that logs do not need to be generated because there are no transaction logs ahead of it or before.

Useful capabilities for tracing today that could translate well for logs too. So long as everthing is stitched together, which is what these tracing libraries are responsible for.

I'm not aware of tail-based (i.e., gathering a batch of logs and sampling the group coherently), but some of the sampling techniques mentioned in the post are absolutely used for structured logging systems.

At least with Honeycomb, for logs unassociated with traces, you can use dynamic sampling[0] to significantly cut down on total event volume and bake in good representivity. And when you have logs correlated with traces, like how OTel does it, you can sample all logs correlated to a trace when that trace is also sampled[1]. We've at least also been doing some thinking about what batching up and coherently sampling "likely related but not correlated by ID" groups of logs. It's definitely in the realm of doable and TBH I find it a little strange that people haven't sought these kinds of solutions at large yet. I think most folks still just assume that the only way you can reduce volume is by giving up some notion of representativeness when that's just not true.

[0]: https://github.com/honeycombio/dynsampler-go

[1]: https://github.com/honeycombio/refinery

> Logs are independent. When you can not store every event, you can drop them randomly. You lose a perfect view of every logged event, but you still retain a statistical view. As we have already assumed you can not log everything, this is the best you can do anyways.

I think one of the google SRE books mentioned why you don't want to simply randomly sample events. If you're do something like storing http events and logging 1 event per 1000 and you have a small burst of errors from some service failing, you'd potentially miss all or most of them. For certain types of events you want a random sample per [endpoint/status code/whatever], potentially at different rates.. .1% of 200 responses, but 1% of 500 errors.

You are correct. I was being a little lazy with my terminology.

You can choose however you want to sample, random just being the simplest and most easily automated mechanism. The key is that you degrade from a perfect record to a statistical record.

Traces should degrade from a perfect record to a statistical record of correlated events (whole traces).

Metrics are “inherently degraded” from a perfect record to a aggregate record for situations where you know you can not have a perfect record, but you want information (even if low precision) on every event.

Yeah, I used to run a service that got a pretty constant 20,000 qps and traces pretty much never showed me anything about the really weird issues. It is nice when you can delay the sampling decision to much later.

(I use zap for logs now and like the sampling algorithm; only duplicate log lines are suppressed, based on the message text but not the structured field values. So if you log "http request finished ok" and "http request errored", then you get a good sampling of both event types. Not distributed, of course, and no guarantee that the same x-request-id will be sampled in other systems.)

> That is an orthogonal discussion and techniques that relieve that bottleneck allow more opportunities to stay on the happy path of "just events with post-processed analysis" that the author is advocating for.

Perhaps Charity's post didn't elaborate on this, but Honeycomb's storage and query engine is designed precisely to relieve the bottleneck you describe and allow for more opportunities along the "it's all just events with post-processed analysis" world described. There are other telemetry backends that allow for this kind of analysis, such as some of Datadog's products (mostly their logging product, to an extent), and several tools that use Clickhouse on the backend. But there's a lot of devils packed into those details, because a column-based database doesn't inherently mean maximal analysis flexibility, even if it is foundational to achieve these things.

More specifically, we decompose traces into events, and re-assemble them on-demand whenever you want to look at a specific trace. Metrics are similarly just events that indeed, when generated, are an aggregation of some measures exported into an event at some regular interval. On the backend you can trivially combine event data whether it is sourced from a trace, log, or metric. The other thing the post talks about that is entirely backed-specific and has no real bearing on how you treat the data, is that every field in your data is effectively indexed, and the cost to process an event with 10 fields on it compared to an event with 500 fields on it is marginal. It's tremendously freeing to treat Observability as a real-time analytics problem space in this way. And when you combine this with intelligent sampling techniques (i.e., not just taking a flat 0.01% of all data or whatever) your costs are very much under control for high volumes of data.

That said, I don't think the post did the greatest job clarifying that "Observability 2.0" as-described is much more about analysis (no pre-indexing, no ingest-only limitations on combining data, no limitations on cardinality) and less about the underlying data (events as the base decomposition of data). The focus on "everything is an event!", event this, event that, etc. is something I've seen lose developers because the abstractions you build atop them (traces, metrics, browser events, etc.) are often some of the most helpful ways to reason about the underlying data, and it's what developers are very often concerned with generating in the first place.

I am no expert but my take was that in many cases, logs, metrics, and traces are stored in individually queryable tables/stores.

Their proposal is to have a single "Event" table/store with fields for spans, traces, and metrics that can be sparsely populated similar to a dynamodb row.

Again, I might have missed the point, though.

My take on the underlying point is that there's a good reason not to use events for every system everywhere. That reason is cost: not cost as in "we can save a bit of money", but cost as in "collecting all our events would double our infrastructure budget". Traces (correlated bundles of events) and metrics (aggregated events, and also other stuff that has nothing to do with events, but let's ignore that) are an attempt to solve the problems caused when you've exhausted the limits of shoving events into a table.

I think GPs point is that sure, you can shove events into a table, but this is just structured logging. It doesn't need to be called observability 2.0 because it's really observability 0.5: the baseline you start with before you need traces or metrics derived from events. All the observability 2.0 hype and the love hearts are a sales pitch from a CTO explaining why you shouldn't shoot yourself in the foot with an unnecessarily heavyweight observability implementation when your log volume is small.

Yep, my post was just saying this is observability 0.5, not 2.0.

However, I actually do agree that you should prefer observability 0.5 in most cases if you have a appropriate implementation.

I am familiar with time travel debugging where you use automatic event instrumentation to stream GB/s per core to fully and perfectly reconstruct all program states. The problems and solutions I see being proposed in the distributed tracing space seem so grotesquely inefficient, yet actionable data anemic in comparison that I am pretty sure most of the limits preventing “just events” are due to inadequate implementations.

However, if you really are running into limits, then it is important to have these known fallback mechanisms instead of just ignoring their existence and calling it a 2.0.

it's weird that you think it's a sales pitch, when i ended it by pleading for other people to share writeups of their similar solutions. i know they exist, and i know people who are desperate to use them.

if it came across as a sales pitch, i def missed the target somehow, apologies.

Apologies if that sounded too cynical. I didn't mean that the general argument sounded "sales-ey", just that a CTO writing about how their company's implementation is better is not taking a thousand-foot objective view of the upsides and downsides.

In this case, I do think 1.0 and 2.0 are flawed names that imply one supercedes the other, when really it's a choice between scalability to levels most companies don't need at the cost of complexity most companies don't want.