Like you said, need to update the reference every time it's moved.
malloc becomes much slower, compared to a GC language (with a compacting GC) where malloc is effectively just a single add instruction.
free also needs to do a lot of bookkeeping while the GC language doesn't need to do anything.
Multithreading introduces another set of issues when dealing with refcounting that is not a problem with a GC.
Reference counting is very predictable. And if you see a bottle neck you can start fixing it.
And with ARC (Swift/ObjC) you get even compiler support to optimize all ref counting administration.
Further malloc doesn't have anything to do with ref counting as ref counting sits on top if it.
Further, GC languages also need to do book keeping: Generations and Compaction.
For malloc, the benefit with a compacting GC is that the free memory will always be in a contiguous segment. Therefore, a malloc operating is similar to a stack allocation in that all you need to do is to move a single pointer. Without a compacting GC, you need to search a tree whenever you want to allocate memory.