The Difference Engine was hand-cranked, but the Analytical Engine was intended to be steam-powered. The thing was going to be the size of a locomotive. Most of that was memory. As I've pointed out before, the big problem in the early days was affordable, fast memory. Babbage's design, at least one version, was to have the ability to store 1000 numbers of 40 digits each. So, 40,000 number wheels, with some kind of mechanism to bring them to the read/write station. Access time would probably have been measured in seconds.
The arithmetic unit wasn't the big part of the machine. It was roughly equivalent to a desktop mechanical desk calculator, after all.
Something similar appears to have happened with Babbage, where he started with a simple calculating device and, thinking about it more deeply, single-handedly came up with a Turing-complete design and was writing programs for it.
Desktop calculators existed long before Babbage. Leibniz built the first mechanical multiplier around 1673. Mechanical arithmetic was known. Babbage's contribution was the instruction decoder and control unit. Mechanical arithmetic was limited more by cost-effectiveness and reliability than by conception. The commercial breakthrough was cash registers, in the mid 1880s. First really cost-effective application. Babbage's machine might have been buildable, but not cost-effective.
A few years ago, there was some guy in the UK talking about an analytical engine build. But he never got very far. I'm surprised someone doesn't have one running in Minecraft or Unreal Engine.