I don't think your solution works.

Suppose N=20. Imagine the server and 19 of the voters are compromised, and are trying to de-anonymize your vote. It "randomly" creates a group composed of you and 19 voters that are collaborating with the server to de-anonymize your vote.

Assuming that the server has M collaborators, it can discover the votes of M / (N - 1) of the citizens.