The real reason Java apps are often associated with being big and heavy is less to do with the technology and more to do with the fact that it's so often used for bespoke enterprise apps where there's no competition and so little incentive to optimise (vs add new features).
I haven't compared the speed of a recent VS vs IntelliJ as I use a Mac now (another benefit JB got from going with Java). But of my complaints about IntelliJ none are really related to speed.
BTW if it's typing responsiveness you're thinking of, I recall that JB had some inefficient drawing algorithms which they've optimised in very recent builds (or it might even be an experimental feature still). It could cause issues on slow graphics cards. That isn't a Java issue though, it was just IJ using inefficient graphics code.