Hacker News Reader

Comment by jitl | Hacker News Reader

Does any OpenTelemetry vendor have a dashboard / graph product in the same level of usability as Datadog?

Honeycomb is decent at what it has, but very limited offerings for dashboard.

Coming from Datadog, Grafana is such a bad experience I want to cry every time I try to build out a service dashboard. So much more friction to get anything done like adding transform functions / operators, do smoothing or extrapolation, even time shifting is like pulling teeth. Plus they just totally broke all our graphs with formulas for like 2 days.

Grafana is to Datadog what Bugzilla is to Linear.

View on HN

As someone used to Datadog I can't find anything that comes close. Datadog makes it so easy to throw adhoc data up on a screen and explore a hypothesis from there. Using Grafana or New Relic feels like going from a 30" screen to a 14" laptop - everything is just harder and more frustrating in ways that are hard to articulate.

Elastic is my choice. Don’t like grafana and dd felt like it was holding your hand while telling you to give them your money as they punch you repeatedly.

I thought New Relic was pretty good but their per seat license cost is super high

They have a SQL like query language that I think can do most of what you're describing

I want improvements like “dropdown menu of tags not limited to 50 items” and “one button forecast on any graph”. Unsure if any SQL query language will solve my annoyance that stuff in Datadog was one click, and in Grafana is 14 clicks or impossible without referencing a Book of the Arcane.

Heck I’m dying for “I can copy a graph between dashboards”. Grafana allows this but if any variable is in the graph but doesn’t exist in the destination, pasting just creates an empty graph.

For Grafana graphs, I think there's a graph library feature that solves that problem but it's been a couple years since I seriously used Grafana so don't remember the specifics

As for forecasting/trends, usually that's offloaded to the underlying data storage system so really depends on where the data is being stored (Prometheus, Influx, Mimir, Elastic, etc)

We migrated from newrelic to datadog (for cost reasons LMAO) a while back and I miss NRQL every single day I'm building a dashboard.

I enjoy having everything instrumented and in one spot, it's super powerful, but I am currently advocating for self hosting loki so that we can have debug+ level logs across all environments for a much much lower cost. Datadog is really good at identifying anomalies, but the cost for logs is so high there's a non-trivial amount of savings in sampling and minimizing logging. I HATE that we have told devs "don't log so much" -- that misses the entire point of building out a haystack. And sampling logs at 1%, and only logging warnings+ in prod makes it even harder to identify anomalies in lower environments before a prod release.

last hot take: The UX in kibana in 2016 was better than anything else we have now for rapidly searching through a big haystack, and identifying and correlating issues in logs.

Datadog log ingest isn't too expensive which will still enrich the logs and can dump to S3 (log archive) where you can use Athena/Trino

Log indexing is $$$, though for sure

Curious on Loki cost. When I priced out ELK at a smaller company it didn't come in much cheaper than $0.50/Gi everyone seems to charge (30 day retention, 2 shards, object storage backups). Back when I worked at JPMC, their internal service was also billed right around there.

I use New Relic and I'm not sure I would recommend it. Sure it was easy to install on all the server but now I keep getting alerts every months (on like the 2nd day) that I ran out of some "quota" of 100GB. I thought that's impossible as I almost don't use it but apparently they (used to) send by default list of running processes every second and that gunks things up.

Also I setup alerts - for >50% CPU or >90% disk full. I do get an alert but it doesn't say which volume or how full is it - what was the actual value that triggered sending the alert. WTF.

That's really a problem with all of them (generating too much data which ends up being expensive) but defaults definitely could have been better/more clear there

Have you looked at https://signoz.io/ ? Its the closest I have tried, but I still wouldn't put it quite at the same level of usability.

Doesn't seem like this can visualize over data stored in Honeycomb like Grafana can :(

It’s a full fledged tool like DataDog which is opensource and can be selfhosted. You can replace honeycomb with it. What features do you think it’s missing compared to honeycomb?

https://github.com/uptrace/uptrace OpenTelemetry-based, feels much closer to Datadog than Grafana.

dash0 is a brand new player, and wants to be "simple", perhaps check it out. Former colleagues of mine so I'm biased, but they do know what they're doing - they built the APM tool Instana, which sold to IBM for $400M.

You can make a tool that’s both powerful and usable. Datadog did it. Adobe did it. There is no downside to usability.

Give IBM Instana a look. It has great otel support. Nice usability. [0][1] Free sandbox for an email [2]

[0] https://www.ibm.com/products/instana/opentelemetry

[1] https://github.com/instana/instana-otel-collector

[2] https://play-with.instana.io/#/home

[disclaimer: I'm an IBMer]

Can this visualize over data stored in Honeycomb like Grafana can?

> Can this visualize over data stored in Honeycomb like Grafana can?

IIUC, Grafana connects directly to Honeycomb via its API to visualize data without storing it. Instana, on the other hand, is a bit different. It needs telemetry data to be ingested into backend before it can be visualized in UI. With Honeycomb, this could be possible if the data can be exported from Honeycomb to Instana.

really? for a lot of us the datadog query language for dashboards is absolute trash. getting it to do what you want sits somewhere between difficult to impossible.

i guess it depends on what you're used to

what do you use instead and what does it give you that you can't get from datadog? as someone who's been locked into datadog for the past few years i'm wondering what i've been missing.

Reply on news.ycombinator.com