Anonymity is crucial to voting, so 'online voting would be easy if we gave up anonymity' would not hold up.
However, cryptography is not necessary just for anonymity, and the extreme difficulty of ensuring the system is neither rigged nor hackable is not just in understanding cryptography.
The way the current voting systems work, each party and many outside observers are going to collaborate on tallying votes in all voting places, and then collaborate in centralizing those votes in progressively larger centers, all the way to the national level. Parties have alternative counts, and they can check their numbers. Since this is a representative democracy, we can trust that the the people we voted for are either convinced the election is fair, or they are collaborating with their political opponents to cheat, which would mean that the election didn't matter anyway.
In an online system, we first need to check that the servers are running the code they claim they are running - not just once, but during the entire electoral process. We need to check that the database of votes contains all of the actual votes that were cast, and no votes that weren't cast, again, not just once, but during the entire process. We need to check that the database used to display results online is the same database used to tabulate the official result. Of course, we also need to make sure the system is highly-available and not prone to DDoS attacks. We need to ensure that someone cutting a few wires doesn't prevent a whole county from voting.
And of course, we need ways to fix all of these problems if and when they do appear. Say you have an entire county claiming their votes are incorrectly tallied. What happens next? How many people will have enough know-how to check whether there was a bug in the system or the people are lying? How long will it take to audit all of the code that ran (this is important because it is delaying the end of the election)?
Not to mention, you wouldn't really achieve that much. Voting will still have to be done from highly secure machines, so you will still have to go to a polling station (it would be absurdly easy to hack people's PC to have them cast the wrong vote). The same goes for checking the result of your vote.
Overall, an online voting system would be as difficult for people to use. It would take less time to tabulate the votes, but probably much more time to resolve disputes. It would be much, much more complex, and the number of people who could understand and check it would go down by at least 1000x. The attack surface would increase tremendously.
Imagine trying to protect your national voting system from the NSA and CIA. How many countries in the world would have the know-how to combat this very real threat? Especially while probably running US-made chips?