You're not really verifying much at all by watching the counting process. It's really just a false sense of security. You actually have no idea if all the ballots are being counted, if they're all real ballots, or if some have been tampered with, etc.
There are millions of programmers in the world, any number of them could decide to audit the code, and if they discover flaw, everyone could be told in an instant. There could be huge security bounties to incentive audits. We trust math and software to maintain all of civilization but for some reason it's impossible to make it work for voting? That seems incredibly unlikely.
Everyone is watching you, if you try to influence the process the police will (and have done) come down like a sack of bricks.
We also (used to) trust OpenSSL... We thought speculative execution was safe too. Paper is easy, paper is cheap. KISS