Blockchain is of course an interesting idea but what I don't really like about it is that someone with more than 50% control can take over the blockchain. While you could see that unfriendly action happening, an adversary could probably create enough chaos to throw doubt on the system. I have been considering an alternative.
In the UK, we have a postal voting system, which can be used by those unable or unwilling to travel to a polling station. You receive a ballot paper and two envelopes to put that ballot paper in. Perhaps other countries have something similar.
You put the ballot paper in envelope A, which is a plain envelope. You then put envelope A in envelope B which has some sort of barcode on it to ensure that there is only a limited number of envelope B. I assume there is a register therefore of those who have voted (just as we have at physical polling stations to prevent double voting).
The B envelopes are torn open and then the A envelopes are gathered. Then the A envelopes are torn open and the votes mixed in with the rest of the votes at polling day.
So another idea might be to double encrypt your voting message. The first message is signed against the key of the verification authority and contains a verification code. The second is signed against the counting authority. The verification system verifies that the vote is valid and hasn't been sent twice and then this is passed to the counting authority which is able to open just the vote message itself without more information.
Drawbacks:
1. We trust the verification authority to not send more votes than it receives
2. We trust the verification certificate and counting certificates are never in the same place. Hardware security modules can help with this.
3. We trust the client-side encryption is correctly implemented
4. We trust that the two authorities have the best interests of democracy at heart
5. We trust that the proposed system isn't just a total lie
So in other words, we trust the same things that we trust with postal votes - that there is a sufficiently decentralised organisation with enough checks and balances and eyes on what is going on to correctly blow the whistle if anything fraudulent is seen.
TL;DR: You need to trust somebody.