Even with competent computer owners it's a problem. Bruce Schneier once pointed out that, given how much money is spent on presidential elections, we should assume that election hackers have budgets of $100 million. They could easily pay for zero-days.

It's not even out of the question that organizations like the NSA could launch attacks, or their counterparts in other countries.

It's also a problem for electronic voting machines without paper trails. One researcher several years ago demonstrated a way to make a virus attacking a popular voting machine. Without a way to audit the results, there's no way to know whether this has happened.