But that applies to everything. Surely you're not that much of a reductionist that nothing people say can move you. "I have never loved you" is also just some vibrations in the air.
The world just is the way it is. One's reaction is the only thing one has power over.