I think the logical incoherence of the Boltzmann Brain saves us from it.

1) What we know about the world leads me to the belief that it's far more likely that a collection of particles in a thermal equilibrium have randomly and temporarily formed into a brain for an instant, than that the entire universe exists as I believe it to.

2) Since I am extremely likely to be a boltzmann brain, everything I know about the world is illusory, and nothing outside of my own mind exists, and my memories don't represent reality.

3) Therefore, everything I thought I knew about the world which led me to believe that I am a boltzmann brain is unlikely to be true.

4) I am unlikely to be a boltzmann brain.