There's some argument to be made about bookkeeping and (maybe? I'm not sure) an extra level of indirection with shared_ptr, but the cost of unique_ptr is literally zero.
And of course no one who cares for their reputation is seen using shared_ptr.
So, the cost of moving a smart pointer onto the stack and, presumably, off again to someplace less ephemeral might matter, on a critical path. But it would be a mistake to exaggerate this: we are talking about a couple of unfortunate ordinary memory operations, not traps, function calls, or atomic synchronizations, when you might have hoped to be using just registers. If you are already touching memory -- something pretty common -- it is just more of that.