Jaeger – A Distributed Tracing System

There are a few things I wish it did, but they are all on the roadmap: http://jaeger.readthedocs.io/en/latest/roadmap/

A friend of mine, Felix Barnsteiner, wrote a profiler for Java based applications, called stagemonitor [1]. He started working on it in 2013 for his masters thesis. Since then, he steadily worked on it in the company as well as in his spare time. Some months ago, he implemented support for distributed tracing. Stagemonitor implements Open Tracing. It also collects frontend performance data, called end user monitoring. They get correlated automatically. And the best thing about it: stagemonitor is free and open source (APL). Get in touch if you have any questions.

[1]: http://www.stagemonitor.org/

A bit of background about how Dapper-style distributed tracing works. Things typically start with an RPC call of some kind (typically from an external source like a public load balancer). At that point, you must decide whether to trace this request or not, which is typically done as a random sample (say, 1% of requests). At that point, the request gets assigned a _trace id_, a random identifier for that request.

The trace id is stored in some request context and propagated to each subsequent service. Each service, meanwhile, divides up its request processing flow into a series of "spans" which represent some piece of computation. For example, a span cover an RPC call or a DB query. Spans are identified by a random _span id_. Once a request has been sampled, all spans for that request are sent to a central span collector where they're stored for later querying.

This model is simple but very limited. It's often hard to know whether a trace is interesting at the outset, hence the reliance on random sampling. For example, you might want to understand why your 99p latency is high, but if you're just sampling 1% the 99p requests will only be 0.01% of your sample.

More generally, interesting events (like errors or slow requests) tend to be rare, and sampling a random, small percent of requests is unlikely to turn up the interesting cases.

A better model, as implemented by lightstep [1] (and an in-progress distributed tracer I've been working on) is to collect all spans. Even with very high request volume it's reasonable to store all spans for at least a few minutes. Doing so opens up all sorts of interesting possibilities, because you can start tracing a request at any point during that window. For example, you might want to trace all requests that have errors in them. Or all requests that take longer than a certain time. Or get a google sample of requests across different latency buckets. Or requests that violate some application invariants you've defined.

Ultimately, though, distributed tracing is so helpful for understanding complex distributed systems and webs of microservices, and it's exciting to see more open-source competition for zipkin.

[0] Dapper is Google's distributed tracing system. The paper (https://research.google.com/pubs/pub36356.html) kicked off a lot of interest in distributed tracing in the broader community. [1] http://lightstep.com/

From the documentation it doesn't look like much, except probably the biggest downside is that you have to add your instrumentation points manually? i.e. there is code change required.

For those who don't know german, Jaeger is "Jäger" is hunter/ranger. A somewhat neutral term in itself but there is also a slight, somewhat remote connection towards some part of the history ("Jagdstaffel" and what not). I have absolutely no idea if this has anything to do with it, mind you - but since the main authors appear to be in the USA, I find that very awkward. Why not just stick to some english name? That would seem a much better it. Or perhaps they think german names are awesome ... it's also weird when you see all the people write Jaeger rather Jäger...