This is exactly what async programs do on the hot path. Consider a 1M request per second process holding 64K of buffers per request. That’s 64GB of allocations per second. Now, assume the requests hit a remote database with 10ms latency. That’s 640MB of live heap in steady state, which ends up in the “long lived” part of most garbage collectors.
Using RAM to save CPU is exactly the wrong tradeoff when such a system becomes CPU bound.
It’s almost always the case that it is CPU bound due to an incoming request spike or elevated retry rates on the backend. Those tend to pile up, creating a 64GB/sec leak.
The alternative is that the system is CPU bound because the heap is large. This is also very common. Unless each collection takes less work as the heap increases in size, backing off the GC rate to free CPU instantly drives the system into metastable failure, where the GC becomes more expensive because the GC is expensive.
Instead of reasoning about this all the time, it’s much easier (for me, granted, I am not a typical java developer) to just jam the CPU intensive work on a low priority event queue so that it uses 100% CPU but never blocks low latency stuff, or things about to retire requests. (Or, stick it in a dedicated but small thread pool if I can’t touch the async event loops).
This ends up being easier to deal with than java, since everything is thread safe, allocations are predictable, and there are CPU escape hatches I can use.
C++ lets me use smart pointers that have exactly the semantics I want, and that are memory safe but racy in practice. Rust makes them actually memory and thread safe, but sometimes adds useless copies, initializations and thread synchronization (or requires unsafe).