Personally, I don't care about the tiny optimizations possible in cases of just using few of them, e.g. Integer, as int[] is an option, and even writing custom maps where the keys are placed the said array ain't difficult.
As for performance, often times I had to PrintAssembly (the article mentions that at the bottom) to ensure the compiler did its bests, e.g. optimizing away boundary checks, inlining calls, etc.