Amateur magician here (but a magician member of the Magic Castle, so I know a few things). Manual dexterity is certainly a useful skill, but it's just a tiny part of the art of magic. You can be a consummate sleight-of-hand artist and still be a mediocre magician, and conversely, some of the best magicians don't use particularly difficult sleights. The best example of that I know of is Dani Da Ortiz's routine on Penn and Teller Fool Us:
https://www.youtube.com/watch?v=5_KcQt0z-eE
Now, Dani is in fact a master at sleight-of-hand, and so one would think that you are watching a masterclass in sleights, but no. There are a few sleights, but they are not particularly difficult, beginner-level. I could do them, and I'm not particularly skilled. The trick is based almost entirely on timing and misdirection and psychological subtleties.
The degree of psychological subtlety in top-level magic continually blows my mind. The best magicians make it look like magic even when you know how the trick is done. Dani's trick is a consummate example. The performance looks like chaos, but in fact every detail is meticulously crafted and serves a purpose. It's almost like watching a dance.