Packet, where are you? – eBPF-based Linux kernel networking debugger

You'll want to have kernel sources handy. In this case you can see the packet ending up in nf_hook_slow function: https://elixir.bootlin.com/linux/latest/source/net/netfilter...

from there you can see only one branch leading to kfree_skb, so you can make a guess that some netfilter "hook" returned NF_DROP. From that you might already be able to make educated guesses that it might be iptables rule, or continue delving deeper to figure out what those hooks are etc

if you have a recent enough kernel, this change https://github.com/cilium/pwru/pull/148 means that it will print the reason the packet was dropped in the output, as logged by kfree_skb_reason - see https://lwn.net/Articles/885729/

There's a whole heap of reasons a packet can be dropped: https://github.com/torvalds/linux/blob/76f598ba7d8e2bfb4855b...

In any case, this makes it less painful than going back to read the kernel source. I don't think that the gif in the README is up to date with this change - it's 2 years old and this feature only appeared 2 months ago.

dropwatch is much more limited, and effectively obsolete. It's a purpose-designed kernel feature with its own netlink protocol that has one serious consumer (the accursed dropwatch.c client). Leave pwru aside for a minute, and everything dropwatch can do, you can do better just with bpftrace.

Dropwatch just tells you where packets were freed from memory by the "bad" path of `kfree_skb()`. It relies on the "good" packet free path to use `consume_skb()` which isn't always true, so you get a number of false positives with dropwatch. It's hard to spot small individual packet loss among all the noise.

pwru offers src/dst IP/port filtering, so it's easier to exclude uninteresting traffic, and it tracks an skb's journey through the entire network stack, printing each function the skb touches over its lifetime.

They are quite different tools, they probably have some overlap depending on situation.

Also the actual dropwatch command is obsolete, it's more common and useful to use the SystemTap script, or perf, or bpftrace.

I can speak for tcpdump and the answer is no. It only looks at the network interface. Something I often want to do is catch which process sent a few UDP packets. Netstat and ss won't catch it because it's too short of a time frame, and tcpdump doesn't contain any information about the kernel.

I've been using opensnitch which uses eBPF rules to track this information lately, but I'm looking for something more flexible.

No. Tcpdump and others are packet sniffers. They clone network packets so the contents can be examined.

pwru on the other hand is used to trace what the kernel is doing with your packets.