It's the same argument people make against many React patterns, that inline objects or inline functions generate additional garbage, but when tested it's found that the time spent cleaning up that garbage is insignificant (and in the React case, workarounds often end up causing performance issues at boot time by trying to move that garbage-generating code out of the render function where it's a non-issue!)
Without measurements you are just guessing, and JITs don't always work the way most will intuitively assume they work. I've tested the overhead of creating and throwing away objects on each iteration of a loop, and the reality is that it just doesn't impact the performance of the code in any meaningful way. The GC happily cleans it up extremely fast, and in some cases the JIT will even compile the code in a way to avoid generating the garbage at all.