Watch Where the Linux Kernel Drops a Packet

I’m guessing this is posted BECAUSE of that article.

Not sure if you are saying weird because you think you, specifically, are unusual, or weird just as a general comment that the brain is weird (like if you wear glasses that make images upside-down, you'll get used to it).

this is a very well known phenomenon, permeating everything, everything around us. it's how advertising works.

dropwatch's definition of drop is literally kfree_skb, ie kernel stops processing a packet (plus changes to drop count at napi device). Don't think of it as recording drops, think of it as recording the ultimate fate of every packet, including e.g. iptables -j DROP, but also e.g. successful transmit by your network driver. dropwatch is a low level tool.

I've found it useful to perform a baseline test to see "normal" drops and their rates, then run intense test traffic and compare.

You pretty much have to understand each reported call site by looking at the kernel source code. As you gain experience with the kernel network code, you learn which locations do what, and how to access the regular statistics for them. Having the (outdated but) relevant kernel networking books on your desk helps.

Note there's also kernel side code for this. Theoretically it's generic, but practically it was written specifically for dropwatch and dropwatch is the only user of it that I am aware of.

https://github.com/torvalds/linux/blob/master/net/core/drop_...

Yes, you can attach an eBPF program to the skb:kfree_skb tracepoint.

For example: perf record -e skb:kfree_skb

Anything the kernel sees, so L2+

This is a standard notation. tcp_v4_rcv+48 means, within the binary kernel image, 0x48 bytes after the start of the tcp_v4_rcv function. You can use the addr2line tool to find the source code line. https://serverfault.com/questions/605946

The tcp_v4_rcv function has two obvious kfree_skb calls, here https://elixir.bootlin.com/linux/v5.6.13/source/net/ipv4/tcp... and a few lines below after "discard_it:" (there may be more if there are #define's or inlined function calls), and without your kernel image and debug info I cannot tell which one the offset corresponds to. Also, clean-up code like kfree_skb is often after a label ("out:") referenced by multiple goto's, and you cannot tell which goto was taken. However, often the function return value contains an error code that identifies it, and you can (often) grab that with perf by attaching a dynamic kprobe to the function exit (it's much easier than it sounds). Or attach a gdb to the kernel (easiest is if the kernel is in a qemu VM) and put a breakpoint on tcp_v4_rcv. There's also the inverse problem of "who called tcp_v4_rcv". Either gdb, or perf record -g, can tell you the stacktrace. (perf is less invasive, so better in production)

As an example, take https://elixir.bootlin.com/linux/v5.6.13/source/net/ipv4/tcp... : if the packet is a SYN belonging to a new connection, but has a bad TCP checksum, goto csum_error, and from there fall-through to discard_it: kfree_skb(skb).

This may sound laborious, and it is, but note that often you don't need to go to this effort. To me, as a troubleshooter, the precise reason might not be that relevant. The function name already tells me this packet has gone up into the TCP receive stack, which (basically) rules out entire problem areas like bridging and routing, tells me if this specific drop is even relevant for me, and/or lets me decide which simpler tools to use next.