To your last point, how is letting someone scan the magnetic stripe less spooky?
Considering that augmented reality app that translates words on signs in real time, I suspect there is far less entropy in the raised numbers on a credit card... though you may be right that the variety of layouts would make it harder than it would at first seem.