Hacker News Reader

Comment by incangold | Hacker News Reader

I think giving metrics and logging a location in a trace is really useful.

But I still dislike OTel every time I have to deal with it.

You can’t do fine grained tracing in OTEL because if you hit 500 spans in a single trace it starts dropping the trace. Basically a toy solution for brownfield work.

...huh? I work with customers who (through a mistake) have created literally multi-million span traces using OTel. Are you referring to a particular backend?

AWS

Well that's a shame, I'm going to ask some folks about that. 500 spans per trace is ridiculously small and I can't imagine any good reason to have that limitation since it's just not that big of a footprint.

OTel doesn't define any limits on the # of spans in a trace (nor the # of attributes on a span!) but it will be bound by the limits of whatever backend you use. In the case of the one I work for, we do limit the total size of a span to be 1MB or less with 64KB per attribute before truncation. Other backends have different limitations. This is the first I've heard of such a small limitation on the total number of spans in a trace though. Traces are just (basically) collections of structured logs with in-built correlation IDs. I can't imagine why you'd limit them like this.

That was two years ago (we tried spans before metrics), so it’s fuzzy. I believe the collector sidecar was fine with it but the backend was not, which complicated debugging. There’s not a clear feedback path in OpenTelemetry that we could find. I completely forgot to mention the tendency toward silent failures. That’s a cardinal sin for telemetry. I would take it out back and shoot it for that fact alone.

The other problem I noticed looking at the wire protocol was that the data for the parent trace doesn’t seem to get sent until the trace closes. That seems like a bookkeeping nightmare to me. There should be a start of trace packet and an update at the end. I shouldn’t have finished spans showing up before the parent trace has been registered. And that’s what it looked like in the dumps my OPs people sent me to debug.

Practically a given outcome, then; we could knock their Managed Prometheus offering off the Internet on the regular. It was just laughable for a company that prides itself in one trillion IAM transactions to 429 some metric ingest

As mentioned by philip below, 500 spans is a very small amount. I have seen customers send 1000s of spans in a trace very easily

This is just not true. We have traces with hundreds of thousands of spans. Those are not very readable but that’s another problem.

How are you storing them, and what do you use to read/visualize/analyze them? I'd imagine just putting them up in a UI becomes a needle-in-a-haystack issue. Are you programmatically analyzing them?

Honeycomb. For shorter traces (most of them), a waterfall view is great. For those long ones, we try to split them up if it makes sense but you can also just run queries scoped to that trace to answer questions about it (how many of the spans are db queries, how many are this query, are they quick, etc etc)

Reply on news.ycombinator.com