Stripping out Maps/Lists with boxed keys and values is the first easy thing to do when perf tuning a piece of code. It's so frequently impactful that it's worth doing immediately without any measurement.

You just swap them out with fastutils or trove and call it a day.

I think you're right, that's probably the best (lowest mental overhead, fewest footguns) way -- always use a List instead of a raw array, and always box primitive types. This also avoids the parallel explosion of Stream types and methods (IntStream, mapToInt(), etc. (and the fact that there's no BooleanStream or ByteStream, etc.)).

That said, I tend to get bitten by the fact that == on 2 boxed numeric values is roughly as misleading as possible -- it does reference comparison, but interns small integer values only, so (Integer)42 == (Integer)42, but (Integer)420 != (Integer)420. (So the small values you probably used in your unit tests will appear to work just fine with your broken, ==-using code.)

ETA: Deleted a misplaced "new". I also don't love that, e.g., ++x does a heap allocation + deallocation if x is boxed (since boxed types are immutable). But usually I care more about the cycles I spend coming up with correct code than cycles the CPU spends executing it.