Hacker News Reader

Comment by avita1 | Hacker News Reader

How do you solve the context propagation issue with eBPF based instrumentation?

E.g. if you get a RPC request coming in, and make an RPC request in order to serve the incoming RPC request. The traced program needs to track some ID for that request from the time it comes in, through to the place where the the HTTP request comes out. And then that ID has to get injected into a header on the wire so the next program sees the same request ID.

IME that's where most of the overhead (and value) from a manual tracing library comes from.

View on HN

100%. Context propagation is _the_ key to distributed tracing, otherwise you're only seeing one side of every transaction.

I was hoping odigos was language/runtime-agnostic since it's eBPF-based, but I see it's mentioned in the repo that it only supports:

> Java, Python, .NET, Node.js, and Go

Apart from Go (that is a WIP), these are the languages already supported with Otel's (non-eBPF-based) auto-instrumentation. Apart from a win on latency (which is nice, but could in theory be combated with sampling), why else go this route?

eBPF instrumentation does not require code changes, redeployment or restart to running applications.

We are constantly adding more language support for eBPF instrumentation and are aiming to cover the most popular programming languages soon.

Btw, not sure that sampling is really the solution to combat overhead, after all you probably do want that data. Trying to fix production issue when the data you need is missing due to sampling is not fun

All good points, thank you.

What's the limit on language support? Is it theoretically possible to support any language/runtime? Or does it come down to the protocol (HTTP, gRPC, etc) being used by the communicating processes?

We already solved compiled languages (Go, C, Rust) and JIT languages (Java, C#). Interpreted languages (Python, JS) are the only ones left, hopefully we will solve these as well soon. The big challenge is supporting all the different runtimes, once that is solved implementing support for different protocols / open-source libraries is not as complicated.

Got to get PHP on that list :)

FWIW it's theoretically possible to support any language/runtime, but since eBPF is operating at the level it's at, there's no magic abstraction layer to plug into. Every runtime and/or protocol involves different segments of memory and certain bytes meaning certain things. It's all in service towards having no additional requirements for an end-user to install, but once you're in eBPF world everything is runtime-and-protocol-and-library-specific.

It depends on the programming language being instrumented. For Go we are assuming the context.Context object is passed around between different functions or goroutines. For Java, we are using a combination of ThreadLocal tracing and Runnable tracing to support use cases like reactive and multithreaded applications.

That’s a very big assumption, at least for Go based applications.

We also thinking on implementing fallback mechanism to automatically propagate context on the same goroutine if context.Context is not passed

I don't think it's unreasonable, you need a Context to make a gRPC call and you get one when handling a gRPC call. It usually doesn't get lost in between.

True for gRPC, but not necessarily for HTTP - the HTTP client and server packages that ship with Go predate the Context package by quite a long while.

Going to be rough for supporting virtual threads then?

We have a solution for virtual thread as well. Currently working on a blog post describing exactly how. Will update once releases

ScopedValue solves that problem: https://docs.oracle.com/en/java/javase/21/docs/api/java.base...

The eBPF programs handle passing the context through the requests by adding a field to the header as you mentioned. The injected field is according to the w3c standard.

Reply on news.ycombinator.com