The part you're missing is secondary effects of designing a language around GC. Most of the time, that means that everything is heap allocated. So even if a GC and RAII-style management did the same amount of work (and they generally don't, GC's often do
less when allocating, for example) while doing allocation/deallocation, non-GC'd languages tend to allocate less in the first place. Additionally, if we're talking about overall performance, indirection can be quite bad on cache locality, so it's not even purely about the speed of allocation/deallocating, but about pointer chasing.
Some GC'd languages also offer tools to manage these problems, of course, all I mean to say is that there are a lot of factors at play here.