What I remember from school (late 90s) is that Babbage was inspired by Jacquard’s punched cards to use then in his Engine, and either this, or Jacquard directly, inspired the same in Hollerith. I don’t recall it as there being a direct line in terms of “modified loom is general purpose computer) but it was certainly an important influence.

In hindsight it seems easy for some of us to make the connection, but, at the time, it must have been quite the breakthrough.