Unpacking Elixir: Observability | Hacker News Reader

Unpacking Elixir: Observability

underjord.io

176 pointslawik15 comments

The Erlang "telemetry" module can even be used "applicatively" with good uses.

An example of that is a "caching reverse proxy" in use at https://transport.data.gouv.fr/, which helps protect data producers servers from excessive load from data consumers (which we manage for them).

"External" requests come to us (via proxy.transport.data.gouv.fr) and if the response is not in cache (for a configured TTL), we forward the request as an "internal" request to the target (3rd party) server.

The proxy uses "telemetry" to generate events for these internal/external queries, without knowing how they will be consumed (https://github.com/etalab/transport-site/blob/master/apps/un...).

The main app receives those events and does what it wants, in this case upserting metrics in the database (https://github.com/etalab/transport-site/blob/master/apps/tr...).

As a result, it's easy to use LiveView to create a real-time dashboard to show these metrics (https://jumpshare.com/s/rdGtTa47CzlcgOz4jsOI - low traffic at time of posting) or generate stats for the data producers which are behind the proxy (EDIT: LiveView source code at https://github.com/etalab/transport-site/blob/master/apps/tr... and https://github.com/etalab/transport-site/blob/master/apps/tr...).

It's all quite natural to do because it's really baked into the runtime!

We use :telemetry to collect usage data per tenant for Supabase Realtime.

We do this for rate limiting but it also makes it very easy for us to attach a listener (https://github.com/supabase/realtime/blob/main/lib/realtime/...) which ships these (per second) aggregates to BigQuery (via Logflare), which then the billing team can aggregate further to display and actually bill people with.

Thanks for sharing!

We just had some orphaned database connections (our bug) to a tenant and were able to find and kill them because of Erlang's observability:

https://gist.github.com/chasers/e4aa33625e6ea39ecb89f03383dd...

edit: link to gist

Not possible in your typical Rails setup where remote console spawns a separate process so you can't inspect what's going on with your actual servers. Even if so, the amount of observability is not remotely (pun intended) close. You don't appreciate it until your production puma workers start going haywire and you have no idea why

You can attach to a running Ruby process with rbtrace easily.

I love this article. I’m about 10 years away from retiring and Elixir/Erlang have been the highlight of my software engineering career because of things like this.

Thanks. Glad it resonated :)

Observability on the BEAM is not an afterthought.

BEAM is actually as good as every new technology tries to pretend to be.

Seriously. It’s an incredible tool and deserves way more praise.

Love this. Great series of articles too. I’m just getting started with Elixir and I had no idea about a ton of this stuff.

> Your compiled C or C++ program will be very different from the code you wrote

Sorry, you just lost beaucoup credibility points. C is just "high-level assembly language". The output looks exactly like the input, modulo some optimizations by the compiler. Every computer programmer knows this. Or should.

Small typo

> their shape very mnuch in compilation

Ah yes. Will correct it to "munch" ;)

Reply on news.ycombinator.com