Once the ballot is in the box, disassociated from the voter then there is no more secrecy concerns.
There are plenty of towns where "everyone knows everyone" and a system that requires trust on poll workers in this aspect is a bit alien to me. For example, one system I am familiar with the ballot itself is not special, the envelope is (it has signatures from the poll workers, which is also problematic because they can potentially surreptitiously mark the envelope in a special way to associate the vote to a specific person), which makes it unnecessary for poll workers to ever be in contact with the ballots themselves.
Overall, my qualms are minimal and am more concerned about the use of computers for the vote than anything else in the process.