A modern "fast" GC has latencies similar to disk I/O on modern storage. We don't make a habit of randomly dropping a blocking disk I/O in the middle of a performance critical path, and disk I/O is less likely to pollute your CPU cache as a side effect than a GC. Given that performance-optimized code is routinely designed to eliminate the overhead of even context switches (e.g. thread-per-core architectures) to great benefit and GC pauses are much more expensive than a context switch even when "fast", it is difficult to develop a theory of how a GC does not have an adverse impact on performance.
You could try to make the argument that most people writing Rust don't care about performance so it doesn't matter, but that is likely to be controversial at the least.