Actually, the government is required to have some people watching the counting (specific rules vary on a per-canton basis). Some states select random citizens for that (similar to jury duty in the US - you get fined if you don't show up, but get paid for your time). Other states have a pool of people they choose from randomly (you volunteer for the pool).
EDIT: Furthermore, Swiss people vote in their community. It's not like you need to go out of your way to watch the counting - it happens in your town hall or (sometimes) the school. It always happens on Sundays (which is a required day off in Switzerland except in some rare cases).