Manual counts aren't practical in the US because our ballots can be crazy long with tons of contests and candidates. It would be very error-prone to do this manually.
If you don't trust the tabulating machines, we have mechanisms to check them at scale. Basically, randomly sample the paper ballots and check them against the machine results. https://en.wikipedia.org/wiki/Risk-limiting_audit
If you don't trust the people running elections, you should know that polling places in the US are run by community volunteers. Literally anyone who's a registered voter can participate and see how the sausage is made. In fact, there's a shortage of these folks, so boards of election are thrilled to get more people participating.
A system based on math, software, and publicly available data would be accessible to independent audit and verification. And could still be backed by a physical system as well, for added security.
Citizens would be much better off not having to trust and be able to verify.
Oddly, even most people that work in technology are incredibly pessimistic about solving technical problems.But anyone doubts that internet voting will be commonplace and incredibly secure in the future is a level beyond pessimistic. The only open questions are when and how.
You're not really verifying much at all by watching the counting process. It's really just a false sense of security. You actually have no idea if all the ballots are being counted, if they're all real ballots, or if some have been tampered with, etc.
There are millions of programmers in the world, any number of them could decide to audit the code, and if they discover flaw, everyone could be told in an instant. There could be huge security bounties to incentive audits. We trust math and software to maintain all of civilization but for some reason it's impossible to make it work for voting? That seems incredibly unlikely.
Everyone is watching you, if you try to influence the process the police will (and have done) come down like a sack of bricks.
We also (used to) trust OpenSSL... We thought speculative execution was safe too. Paper is easy, paper is cheap. KISS
So instead of trusting a few thousands (several thousands, in reality) and a system where you can literally walk in the place where they are counting votes and observe them, you'd rather trust a few tens of people to write the software and audit it? How does that follow?
And this doesn't even take into account the fact that a voting system, as it was mentioned, should be understood by the people voting. Good luck explaining to people what the software and the hardware are doing.
Of course not. All of the software should be open source, with multiple independent implementations, test suites, etc. And the data should be open to analysis by any number of people while preserving people's right to privacy (this is one of the hard parts).
Most people have no any idea how electronic voting machines work today, so why does it matter if laymen understand the internet voting system?
The point is that anyone that is willing and able could verify for themselves that the system does work as intended. Not that most people will actually put in the effort. Most will rely on trusting experts, but anyone is free to verify.
With paper ballots and a handful of officials involved, there is no way for a voter to verify anything, they are forced to rely on trust alone.
Do you really think paper ballots have not been used to rig elections? Because that's absurd given the current situation in Belarus and the long history of ballot tampering going back thousands of years.
Paper ballot counting is security theater. A system based on math and software could be provably secure.