Linux Observability with BPF | Hacker News Reader

Linux Observability with BPF

blog.jessfraz.com

193 pointsdankohn126 comments

As someone who has gone through the ipfwadm -> ipchains -> iptables history, I would generally be pretty meh about another firewalling change. But, I have high hopes that it'll give me the ability to do reasonable traffic shaping in Linux.

I once set up a FreeBSD box to do shaping for an ISP's entire DSL customer base, and it was a couple commands and worked brilliantly! By contrast, I've dabbled with Linux shaping for ~2 decades, and the best I've gotten is one recipe that works well for one limited use case. Maybe it's just me, but I've never been able to make it do my bidding, despite several occasions sitting down and giving it the old college try.

For other kernel paths, I'm really interested in using it to monitor for file modifications in the way that inotify is not very good. And I often resort to strace to figure out low level what is going on, it might be really good for that too.

Looks like an interesting book!

Wondershaper [1] should make it easy on Linux. On *BSD there is AltQ for this which also has PF integration (or well, back when I looked into this in 2004).

I'm curious to hear which use cases didn't work out for you.

Regarding BPF, is it possible to convert IPT rules to BPF for backwards compatibility?

[1] https://github.com/magnific0/wondershaper/

Be wary of poorly-designed wrappers around tc. They often make very wrong decisions: https://www.bufferbloat.net/projects/bloat/wiki/Wondershaper...

sqm-scripts is generally a much better option than wondershaper derivatives: https://github.com/tohojo/sqm-scripts

Thanks for that pointer, I had only used wondershaper as provided in the Ubuntu package, which is version 1.1a, and horribly broken. I recently had a system that was downloading around 1TB of archive data, and killing the network for other use, so I set up wondershaper on that VM. It did indeed limit it, but I couldn't get it to limit it to anything other than the 768kbps it seemed built to do. I really needed more like 50-100Mbps. I'll have to try version 1.4 on github.

That's my impression also, another few decades and linux might get PF, yay... But that is good change, tried picking up IPTABLES multiples times, but always go back to *BSD or dedicated hardware

I like iptables fairly well, though it does take some planning to ensure it can remain speedy. In fact, the only way I could make shaping in Linux work involved using iptables to tag the packets for tc to then shape. Used that for our mirror server, among other things. Even had a fairly slick setup where our core routers would, every few seconds, report their bandwidth usage to the mirror server, which would adjust it's shaping to consume all our committed bandwidth, without interfering with customers traffic or costing us more bandwidth charges.

I think the last thing that's holding Linux back is the root qdisc lock. It becomes very problematic on high PPS workloads and complicated shaper configurations.

Jumbo frames largely solve the problem, but it's the biggest hurdle we've found.

why not use tc for linux traffic shaping? you can even use bpf.

I have never been successful at using tc or the recipes in LARTC or other resources to set up anything more than a basic shaping setup. And even basic ones took hours of messing around to get working. Maybe that's just me?

Not just you. I once spent multiple days trying to get it working, and eventually did - by installing FreeBSD and using pf.

> And I often resort to strace to figure out low level what is going on, it might be really good for that too.

BPF is a double-edged sword in that regard. It's being used to write better tracing tools for normal kernel facilities, but facilities which migrate to BPF become more opaque and unstructured than ever.

> I'm really interested in using it to monitor for file modifications in the way that inotify is not very good

Totally agree. Have had horrible experiences with inotify. I was able to modify https://github.com/iovisor/bcc/blob/master/tools/filelife.py and get a good alternative to find new files.

It still isn't polished and haven't tested it enough so will release it back to community when its ready for beta.

I've been a fan of dtrace for a long time and so I've been meaning to learn BPF to get similar functionality native to the linux kernel. I may pick this book up.

I suggest taking a look at bpftrace ( https://github.com/iovisor/bpftrace ). The features and stability are getting quite good and better over time.

hey! Do you know if anyone is working on getting bpftrace to use BTF yet? Looks like one of the final chunks landed in Linux 5.2

AFAIK not.

See Bpftrace for Linux 2018:

> https://news.ycombinator.com/item?id=18168137

pzakah asks:

> You've mentioned that we do have BTF now in Linux 4.18. I've tried to find if it was leveraged in bpftrace, but it looks like it isn't yet.

Brendan responds:

> That's the old repo (we should add a note to it pointing people to https://github.com/iovisor/bpftrace instead!)

Alastair added struct support for kprobes yesterday, based on the functionality in bcc (which bpftrace uses). That was the final missing piece, and why I'm posting about it now. See the last example here:

https://github.com/iovisor/bpftrace/blob/ma

--- ....

I took a look and according to the last example mentioned they have not added full struct support yet.

Yes, we have an open Pull Request for that: https://github.com/iovisor/bpftrace/pull/734

There’s also a kubernetes version of it:

https://github.com/iovisor/kubectl-trace

Brendan Gregg has a post that provides an overview[1] of the various eBPF wrappers that's worth a look if you want to dive further.

[1] http://www.brendangregg.com/blog/2019-01-01/learn-ebpf-traci...

Thanks!, That looks quite helpful.

It's an advertisement for a book. Cilium is on there and that's a name I associate with blogverts.

There are eleven exclamation marks on that page.

BPF is the new paradigm for synergistic application of strategic opportunities in the use of acronyms.

Thanks for sharing Jessie! I am definitely going to order this book :-)

EDIT: Why did I get down voted? I am not affiliated with this book, the authors of the book, or the OP in anyway. It just so happens that I will find this book very useful, so I thanked the OP for sharing.

It is sad we live in a world where there is so much manipulation that we are suspicious of each other's sincerity. I guess I understand where the downvoter was coming from :-(

This sort of comment isn't looked highly upon on HN as it doesn't really add to the discussion. It's what the upvote button is for.

And in addition, from the guidelines:

> Please don't comment about the voting on comments. It never does any good, and it makes boring reading.

Fair enough. I will remember for next time.

Reply on news.ycombinator.com