Hacker News Reader

Comment by preseinger | Hacker News Reader

OpenTelemetry is moribund and a technological dead-end. Godspeed to any org that builds on it.

I've seen it hyped up like crazy, what's so bad about it and what is the superior alternative?

Instrumentation of a given system is by definition always "larger" than the nominal traffic which flows through that system. That is, given some request R, the possible observable metadata of that request R is essentially infinite. So when you're designing an observability system, the whole ball game is about capturing and constraining the cardinality of that metadata. Observability as a field is _entirely_ about reducing the volume of telemetry data such that it can be usefully captured, aggregated, and queried by operators. That's why we have metrics, traces, and logs as distinct things. They're all optimizations of the same underlying information, each optimized for different consumer use cases.

Concretely: if user U1 does an HTTP GET to /spotify/track/123, that's perhaps 10KB of production traffic, but easily 10MB of telemetry traffic. In effect that telemetry can be modeled as a huge key/val map of metadata, but you can't do it that way and remain efficient, you have to optimize for observability use cases. You have to increment a cardinality-bound set of metric counters for the request outcomes, and maybe emit some best-effort trace data for the request ID, and etc. etc. — _as separate things_!

The engineering costs dominate the design. But OpenTelemetry says this isn't the case. OpenTelemetry says that whatever requirements are on FX00 company CTO feature checklists are valid a priori, and commits to doing whatever is necessary to satisfy them. That's because OpenTelemetry is evaluated not on any technical merits, but on the adoption rate of the CNCF stack among those FX00 companies.

OpenTelemetry is explicitly and exclusively a thing meant to tick off an "observability" checklist item on the checklist of a FX00 CTO's due diligence form. That's it. That's the only goal. Nobody with a choice should be using it. Read the source code, it's abysmal.

Alternatives? Write code that leverages each pillar of observability directly. There's no short-cut. That's the whole point.

> So when you're designing an observability system, the whole ball game is about capturing and constraining the cardinality of that metadata.

If your world is only time series databases that struggle with cardinality, then sure. Fortunately, there's a lot more tools out there, some of which do just fine with high cardinality data.

I don't really agree much with your entire comment. I think you're looking at observability through the lens of 2010s-era tools, and falling deeply into the trap of thinking that this is the only way to do things.

My core claim is that telemetry data for a system is always more than the production data for that system. Yes? No?

Assuming yes, everything else I'm claiming is noncontroversial.

I count about a dozen claims, so I'm not really sure what to begin with. But I disagree with your comment here too.

(shrug) OK. I'm a domain expert here, but that doesn't mean my claims are infallible, of course. Your call. Good luck.

most or all of the observability domain experts i'm familiar with either stopped talking about "pillars" several years ago, or have been actively speaking against the "3 pillars" framing for several years

most of your takes here sound like they're from somewhere around 2016-2018

I don't think anyone opposes that, but I don't think that's your "core claim". Your core claim is that this is a huge problem, one that makes OpenTelemetry "a technological dead-end", and that "the whole ball game is about capturing and constraining the cardinality of that metadata".

That is what it would be nice to have you justify some more. After all, the same could be said of logs, or internal traffic when using microservices/DBMS (request cardinality/traffic will be multiplied).

> Alternatives? Write code that leverages each pillar of observability directly. There's no short-cut. That's the whole point.

I think I am not able to follow you correctly. Is your entire point that auto-instrumentation is too much and one should default to manual instrumentation instead?

You can't delegate "instrumentation" as a monolithic concern to a single authority or package or library like OpenTelemetry and expect to get anything but noise on the other side. And yeah, basically any vendor claiming to do do "auto-instrumentation" is selling snake-oil. Expensive snake-oil, too!

Observability is not something that a vendor can provide without meaningful and deep integration in your infrastructure. You have to do some amount of work, and IMO few vendors deliver value beyond what a single engineer can produce with with a basic internal Prometheus infrastructure + short-term log aggregation.

But OpenTelemetry never claimed to be an auto-instrumentation only library, you can very well just manually instrument your application with the metrics you want and export it your self managed prometheus + log infrastructure. In fact, OTel makes it easier for you if in the future you might want to move to a better self managed TSDB + Log Infra because you know it most likely have an exporter ready with zero effort to re-instrument your metrics or logs

You can definitely define an abstract concept of telemetry which generalizes over use cases, and which can be expressed as a single schema that can be collected generically. The problem is that this definition is too general to be practically useful.

The whole ball game for observability systems is optimization for specific consumption use cases. That _must_ occur at the point of origin, it _cannot_ be deferred. OTel says that it's possible to define a general-purpose exporter for arbitrary telemetry data, and that specialization and optimization of that generalized data can be done later, downstream. This is simply not true.

Could you point me to docs, architecture notes or code where they explicitly say auto-instrumentation is the ONLY approach to using OTel? AFAIK both auto-instrumentation and manual instrumentation are equally supported - https://opentelemetry.io/docs/concepts/instrumenting/

Few of the newer vendors have evangelized the "capture everything, ignore costs" mantra leading people to places where angels fear to tread.

Oh no. Now what should I use? I can't believe something that's actively in development is considered dead these days. Surely there's a better alternative with as good or better community support, right? Right...?

The problem isn't a lack of community, it's that the project is solving a problem that isn't relevant to anyone except Fortune-X00 companies.

Yeah I'm gonna call bullshit on that one. I work daily with everyone from startups and nonprofits to big banks. They use OTel because it solves problems for them. They don't all use it the same way, but they use it.

So, public sector here. We use Honeycomb with OTel and it's most definitely solving our problems. It's incredible being able to see traces up and down our whole stack and pinpoint where an issue is.

What RPS do you serve, net? Everything works to trivial scale, e.g. O(10k) RPS and below. Do you do more than that per server?

You sound like you're trying to have it both ways: "OTel solves a problem that is irrelevant unless you're massive" but also "well of COURSE it can work if you're not massive, how massive are you?"

There are tons of startups and other not-fortune-x00 orgs benefitting from what OTel provides. Your claim that OTel is irrelevant outside fortune-x00 cos is very clearly not true.

I don't agree that OTel delivers value to not-fortune-x00 orgs. It's bloatware and whatever benefits it delivers are more cheaply achieved with other solutions.

Well, we're certainly not Fortune 500 scale, so it works for us.

The person you're replying to seems to have an idealogical bent against OTel. I'll say that "moribund" is about as unrealistic a term as it gets for OTel, given that it's now the #2 CNCF project by activity and still growing.

That said, OTel is far from being easy to adopt still, and despite a lot of us trying hard to change that, it's got a long way to go. If you're using one of the "major" languages (Java, .NET, JS, Python) then it's pretty easy to set up automatic instrumentation and a Collector that you can tee off to your preferred backend analysis tool. But if you need more context from your apps, manual instrumentation outside of tracing is pretty hit-or-miss, and you need to build up a vocabulary around concepts (Resources, attributes, baggage, context, spans, etc.) that isn't easy unless you've got the time to sink into it. It's extremely powerful and has the building blocks to let you capture just about any data you need, pluggable processors and exporters, etc. -- but these building blocks are very unevenly composed into easy-to-use, turnkey-ish components that a lot of people ultimately want.

Reply on news.ycombinator.com