I find it very weird that Xcode is faster than... anything, really

Take some time to learn the navigation and refactoring abilities of the JetBrains family. It is IMHO unmatched by most. Using almost any other IDE feels like a downgrade, especially on large codebases with complex dependencies. Beyond Java, they have excellent support for Python, C#, Go, Javascript, and more. The engine builds a deep understanding of your code, which can make ctrl-space to suggest a method on an object feel delightful (and a Godsend on big Java names).

I also strongly prefer the keyboard shortcut set JetBrains uses. Having to reach all the way up to my function keys for combo operations makes me sad; even when I use other IDEs I love using an Intellij keybinding when available to quickly jump in and be efficient.

But to your point, I also tend to use it on higher end workstations (i.e. minimum of 16GB RAM, local repo pulls on fast SSDs, tons of logical cores, etc.).

They limit the usable memory way too low by default — if you have RAM increase it to at least 2GB, but 4 may be even better and it will be snappy as hell.