There's a tension in platform optimization. Whether it's CPUs or runtimes, people who make platforms tend to look at where they can get the biggest wins for existing apps. So if you bend over backwards to do weird things that get you performance, that weirdness will be exactly what prevents you from getting as big a potential win from future platform optimizations.
With GC in particular, that usually means avoiding the creation of objects that die in middle age. Most GCs are optimized for objects that are ephemeral or long-lived. Ephemeral objects have too few references and not long enough of a lifetime to survive multiple GCs. Long-lived objects live too long to be worth collecting for most GCs. Generational GC uses this to split memory into at least 3 logical buckets (survived 0 GCs, survived at least 1 GC, and long-lived).
Working hard to reuse objects means those objects will become part of the long-lived heap. But reusing them means updating their fields, including their references to other objects, which means they still need to be scanned. So they'll increase the cost of what ought to be cheap GCs of the ephemeral objects; writes to them needs to be accounted, and they need to be included as roots.
The situation becomes even worse if you reuse an object for some time, but eventually find some of them surplus to requirements (the middle age problem).
The costs of designing an application to reuse objects are not to be sniffed at either. It's a reinvention of a typed memory allocator, with all the benefits and drawbacks of manual allocation. Best kept to the implementation-side of properly scoped modules. Don't let the approach pervasively infect the entire app.