Hacker News Reader

Comment by scottlamb | Hacker News Reader

...yeah.

At least I don't have to use all these things. `tracing` itself is essentially mandatory if I want to pick up all the spans and events created by the crates I depend on. But I can (and maybe will...) write my own `tracing::Subscriber`, sidestepping all the various bugs and incompatibilities of `tracing-subscriber` and `opentelemetry{-,otlp,-datadog}`.

(Here's another fun one in `tracing-subscriber`: <https://github.com/tokio-rs/tracing/issues/2519#issuecomment...> These interfaces just aren't right.)

View on HN

I think it's a huge shame, as Rust is one of the few languages that tracing is effectively native in, and it's shocking that the defacto standard of tracing transport doesn't interop well.

Admittedly, I think part of it is just where Rust is on the language adoption curve and where Otel is in its project lifecycle as well. Writing your own subscriber might not be a terrible idea, we internally end up needing to do similar things as we find limitations/bugs that we can't wait for upstream to fix (but we're a vendor and that makes more sense for us than end users!)

Yes. I love Rust, but its observability story is terrible right now. Not just tracing; I was complaining in another thread recently that the async ecosystem doesn't have a production-ready equivalent of thread dumps. <https://news.ycombinator.com/item?id=37792011> I really want to see the whole picture improve (and as I'm able, participate in improving it).

I'm considering writing a tracing subscriber that dumps events, span starts/stops, and span field updates to a terse local log file format. This is a superset of what OpenTelemetry offers. (OpenTelemetry only has the concept of a completed span, which I find really unfortunate.) So I'd write a tool that takes that and pushes it to OpenTelemetry (otelcol-contrib plugin maybe) and more local-focused tools like `logcat`.

Oh huh! What would you do with span start independent of stops?

A tangent on logcat - local observability to me is a really intriguing area, I think there's a story of Otel for local as well if someone can build a good enough local DX for consuming them (we've been told a number of times about this https://github.com/hyperdxio/hyperdx/issues/7 as an example)

> Oh huh! What would you do with span start independent of stops?

* When browsing locally, I'd be able to see spans that never closed (because they were super long-running and/or because the process crashed mid-span). I suppose for the latter case, the otel collector could upload them (marking them as incomplete somehow) when it knows the process shut down.

* When looking at events(/logs), I'd be able to see the current state of all the enclosing spans, including their fields. Easiest to do locally, but ideally also in otel. Maybe some mechanism for automatically copying select attributes from the span to the event for use in otel (details tbd, whether it's selected in code at span creation time, tweaked in the otel collector config, or what).

Reply on news.ycombinator.com