Hacker News Reader

Comment by cbsmith | Hacker News Reader

Oh dear... this seems like a move in the exact opposite direction of what one would hope for. It's basically shoving in structured data into what is at best semi-structured log data, storing all of it, and then using that for things that a tracing system would be better suited to.

I can't help but feel they've learned the wrong lessons from their challenges with tracing.

View on HN

The semi structured nature of logs works to the advantage of Tempo, because as developers we have the flexibility to log _anything_, high cardinality values like cust-id, request latency, gobble-de-gook .. the equivalent of span tags. Instead of indexing these as tags, we get advance search features through a powerful query language landing in Loki (LogQLv2).

But the data starts out structured. It becomes semi-structured when you log it.

I'm telling you, from first hand experience, this does not end well.

There's no reason that your tracing system should not be indexing your tags in an engine that provides advanced search features through a powerful query language.

I agree, if anything the eventual goal should be to invert it. In applications I work on right now, trace tags contain the richest and best-described request metadata. Tags are indexed differently depending on their cardinality, and there is no cardinality limit.

Tempo's implementation seems pragmatic as a short to medium term solution though. Log engines still have a lot more investment and maturity than trace engines. In my work, even though the trace tags contain the best data quality, the tracing system is currently worse at answering a good deal of my questions. It's simply that Splunk has many tools that work well, and the tracing system is behind.

But Jaeger, as an example, will let you choose what back-end engine you want to store your traces in. There is no need to reinvent the wheel just for tracing. You can just leverage what is already out there.

Reply on news.ycombinator.com