Tracing SSL/TLS connections using eBPF | Hacker News Reader

Tracing SSL/TLS connections using eBPF

blog.px.dev

136 pointshtroisi10 comments

bcc comes with the sslsniff(8) tool that does just this, created by Adrian Lopez in 2016. I don't know why the article didn't mention that, since they are using bcc.

It's also a trivial bpftrace program, which we've been recommending for beginners over bcc for years (including in my eBPF summit keynote two years running).

This article would make sense if it was written in 2015.

Brenden, I’m co-opting this comment to thank you for your work in BPF world. Due to you I discovered BCC and bpftrace two years ago. Now as part of my full time work I write BPF programs and applications using libbpf. It’s been a defining part of my career.

Also, sometimes I think it’s a bummer when stuff like this comes up when tons of work has already been done. It’s pretty reliable that unless you have a very specific use-case, there’s probably a BPF tool written to do something.

Thanks for your comment. Yes, it's a bummer to see people waste time reinventing something, and, to then have multiple solutions to the same problem. It's clutter, and wastes end user's time to sort through.

One solution is to spend several years developing expertise and finding the edge of the known world, beyond which is yet to be explored. In this space it means knowing all the tools that are out there. It's no coincidence that I find it easy to draw these diagrams:

https://www.brendangregg.com/Perf/linux_observability_tools....

... and to find new areas where there are missing tools. These skills are connected!

But I'm lucky to have had roles where I have spent over a decade specializing in performance analysis and tooling. For a lot of people with limited time, your best bet is to not learn the entire world, but to learn some narrow specific use case and see what tools exist just for that, and how it can be improved. Just like you say: a very specific use case.

Plus check the open issues on bcc/bpftrace github, since there's stuff there (not just bugs, but tool/feature ideas) that haven't been done yet.

> Yes, it's a bummer to see people waste time reinventing something, and, to then have multiple solutions to the same problem. It's clutter, and wastes end user's time to sort through.

It's only that way if you see the main purpose of this blog post to explain someone how to trace a TLS connection. That could indeed be shortened to 10 lines of "use this existing tool". However to me it doesn't seem like this is the main goal of the post. It mostly uses TLS tracing as an example to describe how eBPF based tracing works, and what components are necessary for it. It can be seen as a tutorial on how to add tracing support for any other userspace function, instead of just being an explanation on how to use a predefined tool to use a single problem. I don't think its a bad thing if more of those tutorials show up.

People learning to code observability tools in eBPF should be starting with bpftrace, and the tutorials should too. I've made this point in blog posts and keynotes etc. Doing it using bcc is how we did it 6 years ago -- it's out of date.

Probably just my lack of understanding of bpftrace, but can it easily be integrated into other tools instead of used interactively / in a script? It seems like this series of posts is about how they built a monitoring agent, so maybe the existing bpftrace scripts aren't practical?

We use bpftrace to feed GUIs at Netflix (json output was added so that it can be incorporated in GUIs). You can come up with situations where bcc is better, but you should really make that clear in the article so as to not mislead beginners.

Amazing strategy. The only thing to watch out for seems to me, is if programs use different TLS libraries or even statically include their own TLS implementation. So in contrast to kprobes, you cannot be sure to catch all traffic with this technique.

(However, of course kprobes and uprobes could be combined to at least become aware of all traffic that did not trigger any uprobe)

Can this also to be used to get metrics and trace events of rejected tls connections and handshakes? I want to know when and why my server rejects a peer and vice versa. Currently this is not visible in a Java application when using the JRE tls lib without AOP and I haven’t been able to make this work at all with OpenSSL.

Really interesting article, thanks!

Reply on news.ycombinator.com