The problem with hand counting is that it scales very poorly. Specifically, the cost of hand counting is the product of the number of ballots times the number of contests on each ballot. US elections tend to have a very large number of contests, which makes the counting very slow. [0] Even with the California 1% manual tally this can take weeks [1] It's true that most of the world does hand counting, but most of the world has one or two contests. It's not unusual for a US election to have 20+ contests on the ballot, which obviously takes 20x as long.
A more scalable approach is to use paper ballots with optical scan followed by a risk-limiting audit [2]. This still provides software independence, but at a much lower cost.
The following blog series on why voting is hard goes into this in more detail: https://educatedguesswork.org/posts/voting1/, https://educatedguesswork.org/posts/voting-hcpb/, https://educatedguesswork.org/posts/voting-opscan/, https://educatedguesswork.org/posts/voting-vbm/, https://educatedguesswork.org/posts/voting-dre/
[0] https://educatedguesswork.org/posts/voting-hcpb/#scalability
[1] https://www.usenix.org/legacy/events/evt08/tech/full_papers/...