Writing High Performance .NET Code (https://www.writinghighperf.net/) has a chapter on this. In C#, time spent collecting depends on the number of still-living objects. That means you want objects you allocate to be short-lived (dead by the time GC happens) or to live forever (they go to the gen 2 heap and stay there). The book suggests object pooling when the lifetime of objects is between those two extremes, or when objects are big enough for the Large Object Heap.
But at the end of the section, the book says:
I do not usually run to pooling as a default solution. As a general-purpose mechanism, it is clunky and error-prone. However, you may find that your application will benefit from pooling of just a few types.