Hacker News Reader

Comment by shanemhansen | Hacker News Reader

To me traces (or maybe more specifically spans) are essentially a structured log with a unique ID and a reference to a parent ID.

Very open to have someone explain why I'm wrong or why they should be handled separately.

View on HN

Traces have a very specific data model, and corresponding limitations, which don't really accommodate log events/messages of arbitrary size. The access model for traces is also fundamentally different vs. that of logs.

There are practical limitations mostly with backend analysis tools. OTel does not define a limit on how large a span is. It’s quite common in LLM Observability to capture full prompts and LLM responses as attributes on spans, for example.

> There are practical limitations mostly with backend analysis tools

Not just end-of-line analysis tools, but also initiating SDKs, and system agents, and intermediate middle-boxes -- really anything that needs to parse OTel.

Spec > SDK > Trace > Span limits: https://opentelemetry.io/docs/specs/otel/trace/sdk/#span-lim...

Spec > Common > Attribute limits: https://opentelemetry.io/docs/specs/otel/common/#attribute-l...

I know the spec says the default AttributeValueLengthLimit = infinity, but...

> It’s quite common in LLM Observability to capture full prompts and LLM responses as attributes on spans, for example.

...I'd love to learn about any OTel-compatible pipeline/system that supports attribute values of arbitrary size! because I've personally not seen anything that lets you get bigger than O(1MB).

Well yeah, there are practical limits imposed by the fact that these have to run on real systems. But in practice, you find that you're limited by your backend observability system because it was designed for a world of many events with narrow data, not fewer events with wider data (so-called "wide events").

OTel and the standard toolkit you get with it doesn't prevent you from doing wide events.

"Wide events" describe a structure/schema for incoming data on the "write path" to a system. That's fine. But that data always needs to be transformed, specialized, for use-case specific "read paths" offered by that same system, in order to be efficient. You can "do wide events" on ingest but you always need to transform them to specific (narrow? idk) events/metrics/summarizations/etc. for the read paths, that's the whole challenge of the space.

You…don’t? This is why tools like ClickHouse and Honeycomb are starting to grow, you just aggregate what you need at query time, and the cost to query is not usually too expensive. The tradeoff is each event has a higher per-unit cost, but this is often the more favorable tradeoff.

> you just aggregate what you need at query time, and the cost to query is not usually too expensive

The entire challenge of observability systems is rooted in the fact that the volume of input data (wide events) on the write path, is astronomically larger than what can ever be directly evaluated by any user-facing system on the read path. Data transformation and specialization and etc. is the whole ball-game. If you can build something directly on top of raw wide-events, and it works for you, that's cool, but it means that you're operating at trivial and non-representative scale.

It does not.

Reply on news.ycombinator.com