Java pioneered this garbage collection stuff because you had cycles of references. You don't need to have cycles. WeakRef is a much better thing now. All you need is reference counting, and you don't need any garbage collection at all. When the reference count reaches 0, you destroy the object and free up its memory. It's far more predictable than GC, too.
And GC isn't "the fastest" to free objects, it has to walk a graph. The fastest is actually arena allocation and then just dropping the whole thing. But that's exactly what owning an entire container of objects can do. If you have a doubly linked list, for example, A[n] -> A[n+1] but also A[n+1] -> A[n] but neither of those should be a strong reference to prevent reclaiming. Instead, the container of that doubly linked list should be the one having a strong reference to its items.