>not only can't do anything about it, but you'll never know.

And what would you do if you are there in person? Challenge them to a duel?

This whole scenario sounds very paranoid to me. In my experience if you treat people decently and perform OK most co-workers won't talk negatively about you behind your back. Or maybe I just never noticed... it did have 0 impact on me though if it happened.