I mean, if you really want to reduce latency the classic solution is to use DPDK: skip the kernel entirely and run in polling mode.

Not great if you care about power consumption, but your latency will be low.