Every commercial job I’ve had has had swap set to zero. We’d rather run out of memory than swap.
The short answer is that not having swap causes the system to run with one hand tied behind its back, since it’s forced to store unused/rarely used stuff in RAM, taking up space that could be used more productively (like disk cache).
As a Java shop, the last thing you want to do is swap. Garbage Collection and Swap tend to be very bad cohabitants.
The only real thing that you may be missing is SSH being swapped out and some other system processes but those may be 2MiB altogether.
Maybe an option would be disabling swap just for the JVM. IDK if that is something you can do with cgroups. But if 99% of your anonymous RAM is pinned anyways there is little to gain.