Voice recorders and storage are very cheap nowadays, keep them on you, running at all times if you ever find yourself in any similar circumstances. That way, it's no longer of "he said, she said", and it's clear who the problem is.
Of course assuming this is legal to do so where you live.