Hacker News Reader

I didn't say such tools don't exist. Honeycomb, mentioned in the post, is exactly Scuba fwiw. I said that over-focusing on traces / metrics and "logs" (in the classical understanding) hides the true power of wide events, and they're not being used widely.

Also IMO open-ended query language doesn't help in quick exploration, it's a barrier. UI and easiness of use is the paramount for adoption. To achieve arbitrary queries one can dump everything to smth like ClickHouse and query it with SQL. This would be a nice addition to any observability stack, to cover for a small percentage of the very deep explorations.

Honeycomb was in fact built to originally be a replacement for scuba, by people who found it so valuable and missed it.

It looks like a "wide-event" is just a structured log, you can send any log containing json to sumo/splunk at it'll parse it out as fields for you. So as I'm understanding it you're advocating for structured logs (which is fine, those are great).

If you want a point-and-click interface to log searching I agree that some percentage of people like to start there (and I think splunk may even have that too), so I'm not opposed to it existing at all, but I feel very strongly that having the more sophisticated capabilities if you want to move beyond a point-click is a requirement.

It exactly is a structured log or log in open telemetry.

To make it easier for myself, I think of spans also as structured logs with a schema that everyone had agreed on, which make it possible to trace requests across multiple services/clients. It's probably more than that, but I don't need academic precision to see how this is more useful during livesite investigations than simply querying logs with unaligned schemas.

Yes, structured log exactly. Why I prefer "wide event" as a term because it has this "wide" component that serves for 2 purposes:

- it highlights the intention of storing as much context as possible - it also hints on the implementation for a system that would serve them. One likely need to use columnar storage to store wide events, there is no way around it

But just a personal preference in the end.

> Just for example, if you wanted to dynamically extract a field at query-time based on the difference of two values and group by that, that's something you can do trivially in splunk/sumo.

You can trivially do that in Scuba too using a derived column, which is supported in the UI. If you need more complex stuff you can write your query in SQL and still use all the supported UI visualizations. And for even more complex stuff, for example if you need joins, the data is usually mirrored to Presto, so you can write arbitrarily complex queries and still visualize the results in the Scuba UI.

I'm not sure if your assessment of Scuba is based on full knowledge of what you can do with it.

I've seen some really nice UI query systems that have plenty of expressive power for 99.9% of queries but also include an escape hatch to allow you to use a query language. Mixpanel has one that I really like and I've seen non-techy business types go to town on it no problem.

The other option I've seen is a guided query language with helpful UI affordances as you type. I've seen it on Datadog as well as Jira's advanced search.

mipsytipsy also worked with scuba, and she specifically designed honeycomb, an observability tool based on wide events, around the motivating idea that a very imperfect tool like scuba could do lots of things better than much more polished and better engineered tools which didn't use the wide events approach.

> if there was a platfom with a powerful enough query-language for it (e.g. give me all requests that go through 4 services, have errors on service 3 but not 4, and are associated with userId 123 on service 1)

Datadog has a new feature for querying the whole trace call graph that they seem to be testing out. Sounds like precisely the thing you're describing.