Hacker News Reader

Comment by BugsJustFindMe | Hacker News Reader

If you get to the end you find that the pain was all self-inflicted. I found it to be very easy in Python with standard stacks (mysql, flask, redis, requests, etc), because you literally just do a few imports at the top of your service and it automatically hooks itself up to track everything without any fuss.

View on HN

> I found it to be very easy in Python with standard stacks (mysql, flask, redis, requests, etc), because you literally just do a few imports at the top of your service and it automatically hooks itself up to track everything without any fuss.

Yes, but only if everything in your stack is supported by their auto instrumentation. Take `aiohttp` for example. The latest version is 3.11.X and ... their auto instrumentation claims to support `3.X` [0] but results vary depending on how new your `aiohttp` is versus the auto instrumentation.

It's _magical_ when it all just works, but that ends up being a pretty narrow needle to thread!

[0]: https://github.com/open-telemetry/opentelemetry-python-contr...

> their auto instrumentation claims to support `3.X`

Semver should never be treated as anything more than some tired programmer's shrug and prayer that nobody else notices the breakages they didn't notice themselves. Pin strict dependencies instead of loose ones, and upgrade only after integration testing.

There are only two kinds of updates, ones that intend to break something and ones that don't intend to break something, and neither one guarantees that the intent matches the outcome.

> Semver should never be treated as anything more than some tired programmer's shrug and prayer that nobody else notices the breakages they didn't notice themselves.

That's precisely my point, but you said it better :).

I have had _mixed_ results getting auto instrumentation working reliably with packages that are - technically - supported.

So recently I needed to this up for a very simple flask app. We're running otel-collector-contrib, jaeger-all-in-one, and prometheus on a single server with docker compose (has to be all within the corpo intranet for reasons..)

Traces work, and I have the spanmetrics exporter set up, and I can actually see the spanmetrics in prometheus if I query directly, but they won't show up in the jaeger "monitor" tab, no matter what I do.

I spent 3 days on this before my boss is like "why don't we just manually instrument and send everything to the SQL server and create a grafana dashboard from that" and agh I don't want to do that either.

Any advice? It's literally the simplest usecase but I can't get it to work. Should I just add grafana to the pile?

Yeah the biggest trouble really is on the dashboarding side of things, not the sending side, and is why there are popular SaaS products like datadog. If you're amenable to saas, datadog is probably the best way. Otherwise, look into SigNoz for a one-stop solution with minimal effort even if there are some rough edges still.

We absolutely have to run it ourselves (...corporate reasons...), it's a lightweight service with only a few hundred users so we haven't had to worry much about perf (yet).

SigNoz does look interesting, I may give this a shot, thank you. I'm a bit concerned about it conflicting with other things going on in our docker-compose but it doesn't look too bad..

Try https://github.com/openobserve/openobserve, it's extremely easy to self-host and it's an all-in-one solution, dashboards included (though admittedly I've seen prettier ones).

Until you run your server behind something like gunicorn and all of the auto imports stop working and you have to do it all yourself.

It works with uwsgi just fine though.

I found this to work fine https://opentelemetry-python.readthedocs.io/en/latest/exampl...

...but with manually running autoinstrumentation in the post fork hook.

I guess there is a lot of undocumented magic in OTel...

Reply on news.ycombinator.com