If you want to use kernel bypass on the server, not the client, then XDP is even more useless. Because on the server world, you have access to professional NICs that you cannot expect clients to have.
It's been decades now that people routinely do kernel bypass on servers using e. g. Solarflare NICs with OnLoad. You just write your networking code normally using BSD sockets API, and LD_PRELOAD the OnLoad SDK which will intercept the socket API and perform a bypass. You get 90% of the latency benefits of kernel bypass without modifying your code, which still runs everywhere.