Is there any way to prevent the observers from knowing who votes? I could see a scenario where a party chooses observers that are likely to intimidate potential voters (e.g. KKK members in a majority black polling station).
If those people still feel unsafe, they are not living in a democracy but under an authoritarian regime. You cannot really have a non-violent, fair democracy under such regime. Democracy isn't just elections. It is creation of bunch of non-elected institutions that guarantee the fairness of the elected stuff. Judicial branch, expert organizations and regulators are all part of it. This has to come from realization that the alternative is violence. Sometimes needed violence. Most resilient democracies in the world like France are direct results of multiple violent events happened because institutions were not capable of striking the balance. Suppressing large swaths of people is just a powder keg. In true democracies, people from all views should have a good mutual understanding that alternative governance systems exist and may even be viable or more stable, but they will be murdering each other and they themselves will eventually be victims to the violence too.
If the police does not uphold the laws that guarantee just elections, if they allow intimidations or treat citizens differently, or if the military tries to influence election results, then you do not have a democracy.