Have you ever tried to label a blank keyboard from memory? Even though you know where all of the letters are and use them all the time without thinking it’s almost impossible.
Interesting effect though. This reminds me that it’s almost impossible to draw a bicycle from memory. https://www.amusingplanet.com/2016/04/can-you-draw-bicycle-f...
I also never use the numeric keypad, but I think that's more to do with being left handed.
I suspect this is some part due to the arrangement of the numbers. Numbers on keypads in these kind of devices often has "1 2 3" on the top row. Whereas on the keybaords and calculators it is reverse. The top row there goes "7 8 9".