Your objects in trees and arrays don't have the same lifecycle?
Quoting GP:
> ... reference counting (RC) have stampeding characteristics similar to GC pauses when releasing large hierarchies e.g. persistent data structures, or large trees of widgets: all the tree is freed synchronously and recursively, if it's large it can be very sensible.
This is mostly where RC & MMM fails. This is also where you are supposed to use arena allocators.
If you have a tree you're constantly adding to / deleting from, you can use a Pool backed by an Arena. At that point you don't use pointers, only indices into the pool. When you want to remove something from the tree, you mark the objects in the pool "deleted" until you want something added back again.