Though if one only uses smart pointers they will have to accept the performance hit that that brings -- in high performance situations, this may be unacceptable.
So, in actual practice, no.
And of course no one who cares for their reputation is seen using shared_ptr.
So, the cost of moving a smart pointer onto the stack and, presumably, off again to someplace less ephemeral might matter, on a critical path. But it would be a mistake to exaggerate this: we are talking about a couple of unfortunate ordinary memory operations, not traps, function calls, or atomic synchronizations, when you might have hoped to be using just registers. If you are already touching memory -- something pretty common -- it is just more of that.