All you need is Wide Events, not "Metrics, Logs and Traces"
isburmistrov.substack.com
isburmistrov.substack.com
> Unlike with prometheus, however, with Wide Events approach we don’t need to worry about cardinality
This is hinting at the hidden reason why not everyone does it. You have to 'worry' about cardinality because Prometheus is pre-aggregating data so you can visualize it fast, and optimizing storage. If you want the same speed on a massive PB-scale data lake, with an infinite amount of unstructured data, and in the cloud instead of your own datacenters, it's gonna cost you a lot, and for most companies it is not a sensible expense.
It does work at smaller scale though, we once had an in-house system like this that worked well. Eventually user events were moved to MixPanel, and everything else to Datadog, metrics/logs/traces + a migration to OpenTel. It took months and added 2-digit monthly bills, and in the end debugging or resolving incidents wasn't much improved over having instant access to events and business metrics. Whoever figures out a system that can do "wide events" in a cost-effective way from startup to unicorn scale will absolutely make a killing.
To be fair, you could probably store and represent everything as wide events and build visualization tools out of that that can combine everything together, even if they are sourced from something else.
The reasons we don't use them for everything are as others in the thread say: it's expensive. Metrics (just the numbers, nothing else) can be compressed and aggregated extremely efficiently, hence cheaply. Logs are more expensive due to their arbitrary contents.
It's all due to expense really.
So I wouldn't say that events are "expensive" while metrics are "cheap" - both depend on the actual implementation, and events can be cheap too.
And so of course if you have to optimise things, you would need to drop some information you pass to the events, but you would need to do the same for metrics (reduce the number of metrics emitted, reduce the prometheus labels,...).
> And so of course if you have to optimise things, you would need to drop some information you pass to the events, but you would need to do the same for metrics (reduce the number of metrics emitted, reduce the prometheus labels,...).
Those are entirely different orders of magnitude both when it comes to size and how much usefulness you lose. In modern storage backends like Victoriametrics a counter gonna cost you around byte per metric per probe. And as you emit them periodically, that is essentially independent of incoming traffic
Capturing the requests into event/trace/whatever other name they gave to logs this month is many times that and is multiplied by traffic.
I thought this argument was about whether wide events can be used for metrics or metrics is a completely different concept. If we want to emulate metrics in events, we would also make them periodically independently of the traffic. Like emit them once in a while. Pretty much like Prometheus scraping works
There are also time-series databases out there that are OK with high cardinality: https://questdb.io/blog/2021/06/16/high-cardinality-time-ser...
There is no doubt that ClickHouse is a super-fast database. No one stops you from using it for this very problem. My point is that specialized time series databases will outperform ClickHouse.
> There are also time-series databases out there that are OK with high cardinality
So does this blog say that tolerance to cardinality means that QuestDB indexes only one of the columns in the data generated by this benchmark?
TSDBs like Prometheus, VictoriaMetrics or InfluxDB will perform filtering by any of the labels with equal speed, because this is how their index works. Their users don't need to think about the schema or about which column should be present in the filter.
But in ClickHouse and, apparently, in QuestDB, you need to specify a column or list of columns for indexing (the fewer columns, the better). If the user's query doesn't contain the indexed column in the filter - the query performance will be poor (full scan).
See like this happened in another benchmarketing blogpost from QuestDB - https://telegra.ph/No-QuestDB-is-not-Faster-than-ClickHouse-...
When cardinality is very high, indexes make more sense.
Btw we use Victoria Metrics now at work. It works good, queries are fast. But we're forced to always think about cardinality, otherwise either performance or cost get hurt. This is okay for the predefined set of metrics & labels and works well, but it doesn't allow having deep explorations.
Here’s an article about this idea: https://datadrivendrivel.com/posts/rmrfprometheus/
You can substitute the Grafana Agents for OTEL collectors as well.
P.S. Prometheus itself can also act as a lightweight agent, which collects metrics and forwards them to the configured remote storage [2].
I think on if the big hills is moving to a culture that values observability (or whatever you choose to call it, I prefer forensic debugging). It’s another thing to understand and worry about and it helps tremendously if there are good, highly visible examples of it.
Edit: Typo.
- some way to get to errors when they happen
- zeroing in on the key performance indicators for your application, and relating them to infra metrics, particularly resources (because cpu, mem, storage, and bandwidth costs money).
Unless you have both domain and infra knowledge, it will be hard to know ahead of time.
For a stateless web app backed by a db, you're typically starting with:
- request metrics (req/s, latency)
- authenticated user activity
- db metrics (such as what you'd get with pganalyze)
It's when there are resource pressure that things get interesting. Here, you have product-fit, you have user traction and growth, but now your app is falling down because it is popular.
It is tempting to just crank things up horizontally and say, you're trying to land-grab users ... but your team will never develop the discipline to develop scalable and reliable software. It's here that you start adding instrumentation to find bottlenecks -- whether that is instrumenting spans, adding metrics, optimizing queries, etc. You also need to craft the dashboard to give actionable intelligence. Here's where Datadog's notebook feature is great -- you explore (and collaborate) with the notebook until you can find the bottleneck, and then export the useful metrics into a dashboard. Then you set up the monitoring, because you have found the key performance indicators.
It's this active search to understand what is going on in _both_ app and infra that shows you the limits of the current architectural designs, guide what you need to do, and validate the architectural and engineering decisions for the future. This active search may involve tools beyond OpenTelemetry or Datadog or Honeycomb -- maybe you have to attach a REPL, or go poking around a memory profiler.
What you _don't_ do is blindly adding these things because having the capability somehow makes things better. Rather, you incrementally improve your capability in order to solve your present scalability and reliability problems with your app and its infra.
This is not to say that wide events aren't worth it - For many things, something like Scuba or Bigquery are invaluable. There's ways to optimize. But we're talking about "One of AWS's largest machines" vs "A couple cores", and I suggest learning Prometheus first.
Haha, since you worked on Scuba I’ll mention IMO this point was by far the biggest flaw of ODS. No one ever performed the metric rollups correctly. Average of averages? And at what granularity? ODS downsampled the older time series data but now perhaps you’re taking a percentile over a “max of maxes”. Except it only sometimes used that method of downsampling automatically.
And I seem to recall the labels “daily”, “weekly”, and “monthly” not being intuitive either, and two of them meant the same thing... that was quite a mess to work with.
A lot of the autoscaling systems were wonky because the ODS metrics they were based upon didn’t represent what people thought they did.
1. They're incredibly cheap to store. In Prometheus, is may cost you as little as 1 byte per sample (ignoring series overheads). Because they're cheap, you can keep them for much longer and use them for long-term analysis of traffic, resource use, performance, etc. Most tracing vendors seem to cap storage at 1-3 months while metric vendors can offer multi-year storage.
2. They're far more accurate that metrics derived from wide events in higher-throughput scenarios. While wide events are incredibly flexible, their higher storage cost means there's an upper limit on the sample rate. The sampled nature of wide events means that deriving accurate counts is far more difficult- metrics really shine in this role (unless you're operating over datasets with very high cardinality). The problem only gets worse when you combine tail sampling into the mix and add bias towards errors/ slow requests in your data.
Not every library or tool supports exemplars, but they're a big part of the Prometheus & Grafana value proposition that many users entirely overlook.
The issue is there are still limits to that, though. I can still get a count of events, or a AVG(duration_ms). But if I have a custom tag I can't get accurate counts of that. And if I want to get distinct counts of values, I'm out of luck. Estimating that is an active machine learning research problem.
People point to Splunk and ELK but they fail to realize that inverted index based solutions algorithmically can’t scale to arbitrary sizes. I would rather point people to Grafana Loki and CloudWatch Logs Insights and the compromises they entail as not just the right model for “wide events” or structured logging based events and metrics. Their architectures allow you to scale at low costs to PB or even exabyte scale monitoring.
Knowing the type of each field unlocks some interesting possibilities. If we can classify fields as STRING, INTEGER, UUID, FLOAT, TIMESTAMP, IP, etc we could store (and transmit!) them optimally. In particular, knowing whether we can delta-encode is important--if you have a timestamp column, storing the deltas (with varint or vbyte encoding) is way cheaper than storing each and every timestamp. Only store each string once, in a compressed way, and refer to it by ID (with smaller IDs for more frequent strings).
It's sickening to imagine how much could be saved by exploiting redundancy in these data if we could just know the type of each field. You get some of this with formats like protocol buffers, but not enough.
Another thing, as you mention, is optimizing for search. Indexing everything seems like the wrong move. Maybe some partial indexing strategy? Rollups? Just do everything with mapreduce jobs? I don't know what the right answer is but fully indexing data which are mostly write-only is definitely wrong.
curious why do you think so? Inverted index can be sharded and built/updated/queried in parallel, so scale linearly.
EDIT: as a user, being able to just run mapreduce jobs over logs is a heck of a lot better experience IMO than trying to torture Kibana into giving me the answers I want.
Another issue is indexing up front is crazy expensive vs doing absolutely nothing but packing and time indexing, maybe some bloom indices. This is really important because the vast majority of log and event and telemetry in general is never accessed. Like 99.99% of it or more.
The technique of something like Loki is to batch data into micro batches and index them within the batches into a columnar store (like parquet of orc) and time index the micro batches. The query path is highly parallel and fairly expensive, but given the cost savings up front it’s a lot cheaper than up front indexing. You can turn the fan out knob on queries to any size and similar to MPP scale out databases such as Snowflake there’s not really much of an upper limit. Effectively everything from ingestion to query scales out linearly without uneven heat problems like you see in a sharded index.
inverted index entry for frequent token can be sharded itself. You can imagine that google doesn't store all page ids in internet for the word 'hello' on the same server.
> This is really important because the vast majority of log and event and telemetry in general is never accessed. Like 99.99% of it or more.
for log processing you are likely correct. I was more wondering in general why do you think inverted index doesn't scale.
I’d note that also Google doesn’t bother keeping a perfect index because perfect fidelity isn’t necessary, unlike in a lot analytic or similar system where replication of ground truth is important. It’s much more important for Google to maintain high fidelity at the less frequent token side of the distribution and very low fidelity at the high frequency side. Logs can’t do that.
Correct, but I think it is solvable problem, if such approach adds value (e.g. one can build and start selling product like that).
Also, complexity may be not that hard. Say, you are building some tiered shards:
- every entry at the beginning is single shard
- once shard reaches 10m records, you split it to 10 shards, which is not that difficult computation, and can be done in XXXms in single transaction
There are definitely products that offer these characteristics. S3 and dynamodb both do, even if you can’t see it. But it took many years of very intensive engineering to get it to work, and they have total control over the infrastructure and runtime behind an opaque api. Elastic search and Splunk are general purpose software packages that are installed by customers, and their data models are much more complex than objects or tables.
I think you mixed two orthogonal topics: you first talked about frequent tokens, and now switched to hot keys(tokens which are frequently queried).
As for frequent tokens, I think I well described algorithm, and it looks simple, and I don't see any issues there, if your metadata store (where you store info about shards and replicas) allows some kind of transactions (e.g. cocroachdb or similar).
For hot keys/shards, as you pointed out, solution is to increase replication factor, but I think if shard is relatively small(10m IDs as in my example), adding another replica online is also fast, can be done in single transaction, and may not require all these movement you described.
A text based format whose sole virtue is descending from a system that was composed mainly of bugs that had coalesced into perl scripts.
It's not the basis of something you could even give away, let alone have people willingly pay you for their agony. Cloudwatch being rather alike in this regard.
I have the impression it does a good job providing visibility tools (search, filter, aggregation...) over structured logs.
Ergonomics is bad, though, with the custom query language and low processing speed, depending on the amount of data you're processing during an investigation.
I don't have any experience with scuba besides this article, but I think you've missed the point. Wide events, based on my understanding, are a combination of traditional logs and something akin to service logs.
This provides two crucial improvements. The first is flexible, arbitrary associations as a first-class feature. As I interpret it, wide events give you the ability to associate a free-form traditional log message with additional dimensions, which is similar to what service logs offer but more flexible. E.g. if you log "caught unhandled FooException, returning ServerException" but only emit a metric for ServerException=1, service logs can't help you.
The other major benefit that you seem to have overlooked is a good UI to explore those events. I think most people would agree that the cloud watch UI is somewhere between bad to mediocre, but the monitor portal UI is nothing short of an unmitigated disaster. And neither give you the ability described in this article, to point and click graph events that match certain criteria. As I read it, it's the equivalent functionality to simple insights queries, except it doesn't require any typing, searching for the right dimension names, or writing stats queries to get graphs.
Why does it have to be an either/or thing? Couldn't you hook up a metrics extractor to the event stream and convert your structured logs to compact metrics in-process before expensive serde/encoding? With this your choice doesn't have to affect the code, just write slogs all the time; if you want structured logs then then output them, but if you only want metrics then switch to the metrics extractor slog handler.
Futher, has nobody tried writing structured logs to parquet files and shipping out 1MB blocks at once? Way less serde/encoding overhead, and column oriented layout compresses like crazy with built-in dictionary and delta encodings.
Bigger than meta? o_0
They complicate things so much that it's sometimes more expensive to observe than to RUN the infra being observed.
And yes, Scuba is in-memory, but it's not the requirement. Check this video out on how Honeycomb implemented their columnar storage: https://vimeo.com/331143124
But I definitely agree with the theme of the article that leaving a big company can feel like you got your memory erased in a time machine mishap. Inside a FANG you might become normalized to logging hundreds of thousands of informational statements, per second, per core. You might have got used to every endpoint exposing thirty million metric time series. As soon as you walk out the door some guy will chew you out about "cardinality" if you have 100 metrics.
It seems like the main motivation for metrics is that sending + storing + querying wide events for everything is cost prohibitive and/or performance intensive. If you can afford it and it works well, wide events is definitely more flexible. A metric is kinda just a pre-aggregation on the event stream.
When you collect any form of metrics, something is choosing that sample rate.
If I transmit a statsd metric representing "CPU usage", I am still sampling it. E.g., I might read the CPU usage every second & generate a statsd stat. That's a sample rate of 1Hz. I have to choose some sampling frequency, since the API most OS's expose is "what's the current CPU usage?".
If the metric is "total number of HTTP requests", then I can definitely just transmit that metric every time I get a request. We're not sampling for that metric.
The latter is inherently a discrete event, with which we can know every data point of, though. Things like CPU, memory, are either fundamentally continuous, or their implementations are simply sampling it.
I do agree the model matters too; Prom's tendency to just poll /metrics endpoints every n seconds means even things like HTTP events are inherently sampled.
In practice this is how everyone does it, but in theory it should be possible to have a non-sampled view of CPU usage (defined as "time process is scheduled onto a CPU"). With the right kernel introspection, you could represent it as a series of spans covering each time slice where the process is scheduled. Perhaps with a concept of a "currently ongoing span" to account for the current time slice.
Do I think this would be more useful than the typical sampled metric? Probably not, outside some niche performance analysis workflows. But my point is that CPU is not actually continuous, and I struggle to think of any metric which cannot be represented without sampling if you REALLY need it.
As for truly contiguous metrics, hmm. How about current battery charge (in Wh)? Host uptime also seems technically continuous (albeit representable by a straight line). (Yes, we track this met; it makes reboots stand out in lieu of my metrics system not providing a vertical marker feature.) Clock drift?
(and I'm going to insert the footnote on this comment about something something Planck units.)
(There are metrics that are continuous, such as CPU usage. Those, yes, you're always sampling.)
You could get pedantic at this point and say that because computers are fundamentally discrete machines, it is technically possible to sample the CPU usage at every tick :p
I'm not particularly fond of those terms; I don't find them descriptive. I don't think they're quite the right terms, either. For example, queue length is fundamentally not a continuous metric: it only changes when the length of the queue does, and if you record those events as they happen, you can get the exact graph of the queue length without there being a sampling frequency. But it is a "gauge" in Prom's language.
But yes, a lot of the metrics surrounding event-like data probably do fall into Prom's "counter".
> sample the CPU usage at every tick :p
Linux has been tickless for years. There's still going to be a time at which the scheduler kicks in, of course, but if the core isn't contested, schedulers these days aren't necessarily going to even trigger. The process on that core can simply run until it sleeps. (Assuming no other process transitions to runnable, and there's no other core available for that process.)
As another poster points out, if we had enough insight into the kernel, though, even still we could get the discrete events of when the scheduler deschedules a core. So, technically we don't have to same. But the practical APIs we're going to use are sampling ones.
- eng observability (traces at Datadog, Sumologic, etc)
- operational research (process mining at Celonis)
- product analytics (funnels at Amplitude, Mixpanel)
As with every new field, there are a lot of different and overlapping terms being suggested and explored at the same time.
We are trying to contribute to the field with a deep fundamental approach at Motif Analytics, including a purpose-built set of core sequence operations, rich flow visualizations, a pattern matching query engine, and foundational AI models on event sequences [1].
Fun fact: creators of Scuba turned it into a startup Interana (acquired by Twitter), who we took a lot of inspiration from for Motif's query engine.
People tend to think kafka is hard, but as you can see from the example, it can be extremely easy.
Many kinds of systems can work great at this scale, which is non-trivial but ultimately not particularly large.
> It's really just 6 servers with 20tb each and another 40tb for backups.
Hopefully you can ingest and store and query O(10k) RPS with an order of magnitude less resources than this?
Sometimes it seems that it's fixable by 'just having a list of people listening', and then you look and all that some of them do is mildly transform your data and pass it along. It doesn't take long before people realize that. 'just logging some events' is making future promises to other teams you don't know about, and people start being terrified of emitting anything.
This is a story I've seen in at least 4 places in my career. Making data available to other people is not any less scary in kafka than it was back in the days where applications shared a giant database, and you'd see yearlong projects to do some mild changes to a data model, which was originally designed in 5 minutes.
As for kafka being easy, It's not quite as hard as some people say, but it's both a pub sub system and a distributed database. When your clusters get large, it definitely isn't easy.
Yeah, I'd always use protobuf or similar rather than JSON for that reason, and if you need a truly breaking change I'd emit a new version of the events to a new topic rather than trying to migrate the existing one in place. It's not actually so costly to keep writing events to an old topic (and if you really want you can move that part into a separate adapter process that reads your new topic and writes to your old one). Or you can do the whole avro/schema-registry stuff if you prefer.
> Making data available to other people is not any less scary in kafka than it was back in the days where applications shared a giant database
It should be significantly less scary: it's impossible to mutate data in-place, foreign key issues are something you go back and fix and reprocess rather than something that takes down your OLTP system, schema changes are better-understood and less big-bang, event streams that are generated by transforming another event stream are completely indistinguishable from "original" event streams as opposed to views being sort-of-like-tables but having all sorts of caveats and gotchas.
> As for kafka being easy, It's not quite as hard as some people say, but it's both a pub sub system and a distributed database. When your clusters get large, it definitely isn't easy.
There are hard parts but also parts that are easier than a traditional database. There's no query planner, no MVCC, no locks, no deadlocks, no isolation levels, indices are not magic, ...
1. Write "logging" data (observability, whatever) 2. Someone else starts using that to drive behavior 3. Change your logging, because it's just logging right? And stuff breaks.
To state it another way, anything you're emitting, _even internal logging_, is part of your API/contract, and therefore can't be changed carelessly. That problem is the same no matter what technology you use.
All you have to do is pass around trace baggage headers, right?
But they aren’t a replacement for metrics because metrics are so god damn cheap.
And while I’ve never used a log system with traces, every logging setup I’ve ever used has had request/correlation IDs to generate a trace because sometimes you just wanna lookup a flow and see it without spending a time digging through wide events/your log system. If you aren’t looking up logs very often, then yeah it seems browsing through structured logs isn’t that bad but then do it often and it’s just annoying…
A tool like splunk can do everything scuba can do and a million things it can't. Sumologic can too.
The reason that splunk/sumologic are so much better than scuba is that they have open-ended query languages rather than this on-rails "only ever do one group-by". Just for example, if you wanted to dynamically extract a field at query-time based on the difference of two values and group by that, that's something you can do trivially in splunk/sumo.
I could write a whole essay on the topic really, but the gist of it is you need a full-scaled open-ended language for advanced querying because 1% of the time you need to do weird stuff like count-by -> a second count-by.
What I will agree with is that traces/metrics do not inherently give you this ability, but absolutely traces could if there was a platfom with a powerful enough query-language for it (e.g. give me all requests that go through 4 services, have errors on service 3 but not 4, and are associated with userId 123 on service 1)
Also IMO open-ended query language doesn't help in quick exploration, it's a barrier. UI and easiness of use is the paramount for adoption. To achieve arbitrary queries one can dump everything to smth like ClickHouse and query it with SQL. This would be a nice addition to any observability stack, to cover for a small percentage of the very deep explorations.
If you want a point-and-click interface to log searching I agree that some percentage of people like to start there (and I think splunk may even have that too), so I'm not opposed to it existing at all, but I feel very strongly that having the more sophisticated capabilities if you want to move beyond a point-click is a requirement.
To make it easier for myself, I think of spans also as structured logs with a schema that everyone had agreed on, which make it possible to trace requests across multiple services/clients. It's probably more than that, but I don't need academic precision to see how this is more useful during livesite investigations than simply querying logs with unaligned schemas.
- it highlights the intention of storing as much context as possible - it also hints on the implementation for a system that would serve them. One likely need to use columnar storage to store wide events, there is no way around it
But just a personal preference in the end.
The other option I've seen is a guided query language with helpful UI affordances as you type. I've seen it on Datadog as well as Jira's advanced search.
You can trivially do that in Scuba too using a derived column, which is supported in the UI. If you need more complex stuff you can write your query in SQL and still use all the supported UI visualizations. And for even more complex stuff, for example if you need joins, the data is usually mirrored to Presto, so you can write arbitrarily complex queries and still visualize the results in the Scuba UI.
I'm not sure if your assessment of Scuba is based on full knowledge of what you can do with it.
Datadog has a new feature for querying the whole trace call graph that they seem to be testing out. Sounds like precisely the thing you're describing.
> Structured Log which is pretty much the Wide Event
The OP's point is that you can just capture anything that one might call a metric/span/trace in a structured log.
So, it's a bit more specific, in some senses: a "metric" is a structured log bearing some value. E.g.,
{
"@timestamp": "…",
"query_kind": "list_people_in_org",
"query_time_s": .051
}
ELK stacks let you visualize such structured logs into graphs with only moderate pain.
A "span" is a structured log with start/end timestamps.
A trace is just a tree of spans, built out of many structured logs. In a prior life, we'd have,
{
"span_id": "root_span_id/subspan_a/subsub_span_b",
…
}
I think where I depart from the OP: it takes too much space. The OP advocates sampling; I'd rather a specialized data structure for metrics that can store a data point efficiently. I.e., at least in sizeof(timestamp || f32) bytes, ideally better, and then I can just not worry about sampling. (In a sense; if you're watching a continuous value, you're always sampling. But for discrete events that you're recording, I can just record all the events. But I'll only get a number. Sometimes, that's okay. I feel like this is a bit spiritually different than the sampling the OP is discussing.)
Structured logs are ... well, logs that are structured (probably json) and machines read 'em to offer nice analysis capabilities. We've largely as an industry decided that these are great, and we should use them instead of unstructured logs if we can.
But just because a log is structured it doesn't mean it has what you need to effectively debug an issue! You usually need app-specific data added as key-value pairs to that log so that you can correlate stuff like "stuff is slow on this endpoint" with "and these are the device versions and user_agent strings that correlate the most with that".
And a lot of developers are used to dumping some of that data into random logs, but without making that data a part of a "wider" structured log, it can be really hard to correlate the behavior you don't like with other data that aids with debugging.
Hence, a desire to call them "wide events".
Anyways, I'm not particularly sold on that term either. But I do think there's some need to describe not only structured logs, but structured logs that contain all the rich info you need to debug most stuff in production with general ease.
JSON, or usually ndjson, is just simple, and widely supported. It's not the only format, nor even the best format. But it is easily produced, and better than no format at all.
> But just because a log is structured it doesn't mean it has what you need to effectively debug an issue! You usually need app-specific data added as key-value pairs to that log so that you can correlate stuff like "stuff is slow on this endpoint" with "and these are the device versions and user_agent strings that correlate the most with that".
Yes, you have to instrument the app. There's no getting around that. Experience will tell you what to log.
> And a lot of developers are used to dumping some of that data into random logs, but without making that data a part of a "wider" structured log, it can be really hard to correlate the behavior you don't like with other data that aids with debugging.
Again, experience. Log correlation is easier with what the article calls a "SpanId" or a "TraceId", some places call it a "RequestId" or a "CorrelationId", just some way to say "all longs forming a particular request" so that you can just read a request, from start to end. A simple thing … but you have to log it, or you won't have it.
Agreement across services about naming it all the same thing in your log store of choice also helps, so that if you need to trace it across services, you can. This really isn't hard, but it depends IME on how well your organization is otherwise functioning. E.g., do engineers talk & plan? Can they say "X would help with Y" and then have people just go "yeah, it would. Done." or is it a long drawn out fight because technical leadership can't fathom basic stuff like what a RequestId does.
> But I do think there's some need to describe not only structured logs, but structured logs that contain all the rich info you need to debug most stuff in production with general ease.
I feel like you're just saying "log the data we actually need", which isn't a terribly actionable thing if you don't already know the answer. If I had to make a recommendation: outcomes & latencies of I/O.
Otherwise, put something into production, & be on-call for it.
I've never personally heard it, but it the context of the OP / metrics/logs/traces, it's a pretty self-descriptive term.
Why I prefer "wide event" over "structured log" as a term because it has this "wide" component that serves for 2 purposes: - it highlights the intention of storing as much context as possible - it also hints on the implementation for a system that would serve them. One likely need to use columnar storage to store wide events, there is no way around it
But just a personal preference in the end.
I personally think the challenge is in passing around the metadata.
In the end anything can be represented as a structured log.
A span is NOT what the OP calls a “system wide event”. A span has a begin and end time. What he/she describes doesn’t have that.
In the end giving different kind of instrumentation instruments a name makes sense, mainly for processing them / rendering them / altering on them.
The ELK stack is a popular choice, albeit with a focus on search rather than OLAP.
If SaaS is an option, a simple staring point in AWS might be Data Firehose into S3 with Athena. Snowflake can load and query the data too. All of these tools have multiple frontend options with a proportional relationship between cost and user-friendliness.
I honestly just do this in PostgreSQL until my project outgrows it. Create a table with a JSONB column and as few indexes as possible to improve write throughput. Cover a timestamp column with a BRIN index to filter by date range.
> Tempo can ingest common open source tracing protocols, including Jaeger, Zipkin, and OpenTelemetry.
Grafana agent as OTEL collector on the application hosts, Grafana Tempo as backend for traces, Loki for logs and Prometheus for Metrics.
The cool thing about Tempo it generates metrics for ingested spans and their labels (spanmetrics) so this allows us to explore “unknown unknowns” as the author calls it in a very cost efficient way.
I keep hearing that this doesn't scale. When I worked at Google, we used this sort of system to monitor our Google Fiber devices. They just uploaded their logs every minute (stored in memory, held in memory after a warm reboot thanks to a custom linux kernel with printk_persist), and then my software processed them into metrics for the "fast query" monitoring systems. The most important metrics fed into alerts, but it didn't take very much time to just re-read all the logs if you wanted to add something new. Amazingly, the first version of this system ran on a single machine... 1 Go program handling 10,000qps of log uploads and analysis. I eventually distributed it to survive machine and datacenter failures, but it ultimately isn't that computationally intensive. The point is, it kind of scales OK. Up to 10s of terabytes a day, it's something you don't even have to think about except for the storage cost.
At some point it does make sense to move things into better databases than logs; you want to be alerted by your monitoring system that 99%-ile latency is high, then look in Jaeger for long-running traces, then take the trace ID and search your logs for it. If you start with logs, you have that capability. If you start with something else, then you just have "the program is broken, good luck" and you have to guess what the problem is whenever you debug. Ideally, your program would just tell you what's broken. That's what logs are.
One place where people get burned with logs is not being careful about what to log. Logs are the primary user interface for operators of your software (i.e. you during your oncall week), and that task deserves the attention that any other user interface task demands. People often start by logging too much, then get tired of "spam", and end up not logging enough. Then a problem occurs and the logs are outright misleading. (My favorite is event failures that are retried, but the retry isn't logged anywhere. You end up seeing "ERROR foobar attempt 1/3 failed" and have no idea of knowing that attempt 2/3 succeeded a millisecond after that log line.)
For the gophers around, here's what I do for traces: https://github.com/pachyderm/pachyderm/blob/master/src/inter... and metrics: https://github.com/pachyderm/pachyderm/blob/master/src/inter.... If you have a pipeline for storing and retrieving logs (which is exactly the case for this particular piece of software), now you have metrics and traces. It's great! I just need to write the thing to turn a set of log files into a UI that looks like Jaeger and Prometheus ;) My favorite part is that I don't need to care about the cardinality of metrics; every RPC gets its own set of metrics. So I can write a quick jq program to figure out how much bandwidth the entire system is using, or I can look at how much bandwidth one request is using. (meters logs every X bytes, and log entries have timestamps.)
I think since we've added this capability to our system, incidents are most often resolved with "that's fixed in the next patch release" instead of multiple iterations "can you try this custom build and take another debug dump". Very enjoyable.
To me the golden rule is "show your work". Every operation that can start and end should log the start and the end. If your process is using CPU but not logging anything, something has gone wrong. Aim to log something about ongoing requests/operations every second or so. (This is spammy if you're doing 100,000 things concurrently. I use zap and zap's log sampling keys on the message; so if your message is "incoming request" and 100,000 of them are arriving per second, you can have it only write the logs for one of them each second. I hate to sample, but it's a necessity for large instances and hasn't caused me any problems yet.)
I also like to keep log levels simple; DEBUG for things interesting to the dev team, INFO for things interesting to the operations team, ERROR for things that require human intervention. People often ask me "why don't we have a WARN" level, and it's because I think warnings are either to be ignored, or are fatal. Warnings ("your object storage configuration will be deprecated in 2.10 and removed in 2.11, please migrate according to these docs") should appear in the user-facing UI, not in the logs. They do require human action eventually.
Overall, I'm more of a "print" debugger than a "step through the code with breakpoints" debugger. To me, this is an essential skill when you're running code on someone else's infrastructure; you will be 1000 times slower at operating the debugger when you are telling someone via a support ticket which commands to run. (Even if the servers are yours, I don't love sshing into production and mutating it.) So ultimately, the logs need to collect whatever you'd be looking for if you had a reproduction locally and were trying to figure out the problem. It's an art and not a science; you will get it wrong sometimes, and your resolution for the underlying bug will include better observability as part of the fix. This is usually enough to never have a problem with that subsystem again ;)
Can pretty easily achieve this with structured logging in GCP with their metrics explorer. Pretty cheaply I might add. Sentry can also do a bit of this if you're on something like fly.io (they offer a year free).
I don't think either would completely replace tracing in a complex system for me. At least not in the context Ive worked.
SELECT SUM(1 / samplingRate) FROM AdImpressions WHERE IsTest = False
How does tracing? Are folks adding PII to spans? I suppose you could but I'm not sure why.
https://colab.research.google.com/drive/1Y65qXXogoDgOnXFBDyF...
> Yet, per my own experience it’s still extremely hard to explain what does Charity meant by “logs are thrash”, let alone the fact that logs and traces are essentially the same things. Why is everyone so confused?
Charity is not confused, Charity is incentivized. What she means by "logs are trash" is "I do not sell a logging product". (and, to be clear, I'm only naming Charity individually here because that's who the author named in their article.)
> When I was working at Meta, I wasn’t aware that I was privileged to be using the best observability system ever.
The observability system that is appropriate for Meta is not necessarily appropriate for your project. Those tools are cool but also require a pretty serious investment to build and operate correctly. It's very easy to wade into a cardinality explosion problem when tagging and indexing everything you can imagine, it's very easy to wade into problems regarding mixed retention policies when some events are important and others are less-important, it's very easy to wade into a latency-sensitivity issue if you're building a log/event collection infra that you don't allow to ever lose data, etc. As it turns out, observability is a large topic.
The idea that there's one "best" way to do observability is a little ridiculous. Like when I worked at Etsy some of the data was literally money, when I worked at Jackbox Games we made fart joke games (Quiplash, Drawful, Fibbage, You Don't Know Jack, etc) and the infrastructure was nothing but pure cost. The observability needs of those two orgs were phenomenally different, because the products were different, the revenue models were different, the needs of the users were different, etc.
Also this notion that "all you need is wide events" is the answer seems ... really shallow. A data point is an unordered set of key-value pairs? That's how ... a LOT of logging, metrics, and tracing infra expresses things at the level of an individual record/event. The difference is in the relationships between the keys and values, the relationships between the individual records, etc.
and "stop sampling" is just a bizarre marketing angle. If you have 1 million records or 10 million records and you get the same squiggly line out of analyzing it, congrats you have inflated the size of the data that nobody ever looks at. There is only one person who this benefits and it's the person who charges you for the pipeline, which is exactly why people who sell a pipeline are incentivized to tell you that sampling is bad: if you are sampling, you are sending and storing and querying fewer data points, so they are charging you less money. They are getting paid to tell you that sampling is bad. Sampling is not good or bad, sampling is sampling. The reality is that in a lot of these systems, the vast majority of the information will never, ever be looked at or used. Whether or not that matters is entirely context dependent.
Wait, where did I mention stopping sampling? :) The opposite: the article is praising the native sampling Scuba has.
> There is only one person who this benefits and it's the person who charges you for the pipeline, which is exactly why people who sell a pipeline are incentivized to tell you that sampling is bad
I largely agree, and I'll say that at least with Honeycomb (since it's mentioned by the author) we make sampling a key component of pretty much any deal before anything gets signed. For small stuff this clearly doesn't matter so much, but it basically boils down to:
- Most of your data is probably uninteresting because it's uniform and represents success cases, so you just need a statistically significant sampling to get a sense of what "okay" means for comparisons
- Whenever there's an error or high latency, there's almost always something interesting in there, and you probably want all of it
And so this typically works out to generating an order of magnitude or two more data than you actually need to get an accurate view of what's going on any any point in time.
And so when you do this, you can (and probably should?) pack your events/logs/traces/whatever-you-call-it with a bunch of data.
There's some examples where you can't do this, though. Some people want to be able to do something like plug in a customer ID and dig up the exact trace that represents something they complained about. Or there's some compliance to adhere to, legal or non-legal, where it's still less money to pay for everything unsampled than it is to deal with the consequences of non-compliance. But for most organizations I'd say what I mentioned above holds true.
...but that's just one of the several ways that some folks will frame up Observability. The term has been kleenexed a bit so it now means whatever any vendor says it means, and they all say varyingly different things.
And there are quite a few solutions on top of it.
A couple of examples that seem to be interesting (however I didn't use them in real life):
ClickHouse is replacing Elasticsearch in this context, and is providing the same data storage, but with better compression, and not holding data in memory?
Is that correct?
Don't have a good comparison at hands though, but a few random links on the topic:
https://engineeringat.axis.com/schema-changes-clickhouse/ https://clickhouse.com/blog/contentsquare-migration-from-ela... https://clickhouse.com/blog/100x-faster-graphql-hive-migrati... https://blog.zomato.com/building-a-cost-effective-logging-pl...
We successfully replaced Elasticsearch with ClickHouse within a Year. We went from having difficulties managing 3 months of data to storing 5 year+ of data. We also had difficulties onboarding users into Kibana and Elasticsearch world, this was not such a big problem with ClickHouse since most developers feel comfortable with SQL.
This change together with Apache Superset for the BI layer made a huge impact in the amount of internal users that could extract value from the data collected. Went from around 150 to 800 internal users.
We have not yet managed to bring everything into a wide event table but as long as the logs can be joined with the metrics under an interface that the developers feel comfortable with and the solution is cost-effective enough to allow all relevant context to be collected you will get far imo.
> This change together with Apache Superset for the BI layer made a huge impact in the amount of internal users that could extract value from the data collected. Went from around 150 to 800 internal users.
Really impressive results.
OTel Events are just OTel logs with a name. An OTel log is a log body with a trace ID, span ID, severity, etc. But that's also an event as per the author's definition.
In OTel, a span is just a structured log, which is also an event (as per the author's definition of event). So is a Span Event, which is a log-like entity that you can produce in the context of a span. And OTel metrics produce what are called "metric events", which are also events.
IMO it's one of those things that's horribly confusing until one day it's not, and then everything starts looking like how the author described.
Something with a real meaning that is enables a step-change is development practices. Adoption is organic initially because the pain it solves is very real.
But as awareness of the idea grows it threatens established institutions and vendors, who must co-opt the concept and redefine it such that they are included.
If they can’t be explicitly included (logs, metrics, traces)[0], then they at least make sure the definition is becomes so vague and confused that they are not explicitly excluded[1].
Wide events and a good means to query them covers everything, but not if you as a vendor cannot store and query wide events.
[0] as the article notes, one of these is not like the other. [1] Is Scrum Agile? What do you mean a standup can’t go for an hour? See also DevOps as a role.
For everyone else, more specific data structures, sampling and careful consideration of what to record are essential.
I own a system in one of the tech giants where sampling is counterproductive to our problem domain, and where in the pipeline we deal with each of those issues is our bread and butter problem that can sometimes swing costs by $millions.
> Also note that we have never mentioned anything about cardinality. Because it doesn’t matter - any field can be of any cardinality. Scuba works with raw events and doesn’t pre-aggregate anything, and so cardinality is not an issue.
This is how we end up with very large, very expensive data swamps.
Put another way, you're in charge of a large telemetry event sink. How do you incentivise the correct sampling behaviour by your users?
https://docs.honeycomb.io/manage-data-volume/refinery/sampli...
At least in GCP you can apply a filter to prevent ingestion and set different expiries on log budgets. This can help control costs without missing important entries.