The things I mentioned (local symbol auto-complete, reference-chasing, global go-to-symbol) are things that modern code editors do by default for any programming language they can syntax-highlight. (And also, unlike with an IDE, they don't require any understanding of a particular "project" format to do so. You can git-clone any random repo, throw it at them, and they'll see a directory tree of source files and blindly index everything in it.)
> ask the IDE to do a refactor of some code, remove unused dependencies,
Honestly, I think I just... haven't ever felt the need to do these things in Java?
When I'm fixing bugs, I'm usually just rewriting one line that calls something to call something else instead. Or adding a check for an edge-case. Or wrapping a blocking call in a semaphore or worker pool. Or making something more idiot-proof by replacing a primitive `int numSeconds` with a Duration. 99% of my "coding" time for a Java problem is actually spent investigating whatever heisenbug brought the problem to me (usually after already passing through a coworker's IDE's debugger, to no avail.) The actual solution I arrive at after such investigation, is usually trivial.
Which is to say, I write Java code, but I never really need to maintain any. I'm not on the team that owns the codebase; I'm just a pinch-hitter there to quickly fix customer-facing DevOps issues discovered in prod.
And, at least for that role — which might admittedly be an unusual one for a Java project — I don't think an IDE would help me get problems solved any faster than I already solve them.