Computers and looms have always had a close-knit history (sorry I had to do it). During the industrial revolution there was a ton of innovation in looms, because producing fabric was very labor intensive. These early industrial looms are just sophisticated mechanical computers.
I believe that the reason early computers used punched cards, was because it was a technology already widely used for looms. Babbage just looked for what was already available for his Difference Engine.
What's interesting is that this idea has (very loose) roots in antiquity. Many beliefs had deities of fate that controlled the destinies of mortals by weaving.
The guy who showed it could be done automatically was Jacquard who built an automatic loom directed by punched cards, patented in 1804. Hollerith later took the idea but never acknowledged Jacquard's massive contribution.
John Von Neumann's dad brought home loom programming cards too.