You've missed the point of Boltzmann brains. No one who studies this stuff thinks we're a Boltzmann brain. The question is whether we can rule out theories that predict Boltzmann brains as the numerically dominant observer with our observations. Indeed, many very popular theories of cosmology (e.g., eternal inflation) are in apparent conflict here.