Atomic reference counting does incur a slowdown.
If your issue is "atomics are slowing down my app", then I assume you already milked the code to the latest micro second of performances everywhere else. This is likely not the case here and not a general advice I would give to anyone.
Remember, some dev in python where concurrency is inexistent, start-up time is horrendous and performances are abysmal compared to rust. (this is exaggerated: of course you can run stuff in parallel in python)
Once you have an Arc<T> you can `as_ref()` to get a &T. So maybe you need an Arc to share something with another thread, so you clone it once. Once you're in that thread though you can go back to just using `&` and never touch the atomic again.