Problem is memory locality, or lack thereof. Having to chase pointers around and not having things laid out in sequence doesn't play well with the CPU cache.
There have been garbage collectors designed to improve locality of predictable accesses, but they are not the norm.