I think you're probably agreeing with Carroll's argument about cognitive instability.
Our universe, assuming we aren't hallucinating or falsely remembering our experiences in it, likely allows for Boltzmann brains that hallucinate or falsely remember a universe like ours, and also those that hallucinate or have false memories about universes very different from ours. I can't even guess at the probability distribution of false experiences of one type complex universe versus another type of complex universe, but hallucinating any universe well is a lot less probable than a lump of Boltzmann grey matter that does not have any memories of experiences (false or otherwise) at all, because a thinking, remembering brain (Boltzmann or not) has a much lower Boltzmann entropy than a non-thinking, non-remembering brain.
> I don't see how the properties of our Universe give any information on whether Boltzman Brains are possible
It's just fluctuation theory combined with two things: a cold and nearly uniform photon "gas" in the far far future assuming expansion continues, and the ability of overdensities of even IR photons to combine into more complicated bound states of matter. The cosmic microwave background exists, and there will also be an even sparser gas of ions that can capture any charged leptons produced if enough CMB photons localize and interact. If forming dense collections of molecules this way seems highly improbable to you, you are on the right track. :-)
The idea is to try to develop a no-go theorem about the early universe being in a high-entropy condition. Fluctuation theory does very well with structure formation, but if it's just fluctuations in an effectively uniform hot gas of matter (in the most general sense) then why do we have a dust of galaxies (at least one of which has real brains) rather a dust of Boltzmann brains? The latter are much much more likely than the former to fluctuate out of a higher-entropy state.
Our past being much lower entropy even at the hottest densest phase of the universe solves some arrow-of-time / manifold-orientability problems as well.
Among other things this motivates attempts to observe the "dark ages" after the CMB formed but before the first starlight, and detailed studies of the fine-grained structure of the relic fields (of which the CMB is one) produced in the early universe. The "noise" in the CMB is expected to be (and frankly appears) at lower entropy than the "noise" in the distribution of matter. But the "noise" in the latter does not appear to produce signs of Boltzmann brains in (or nearby) high-redshift galaxies. (Intriguingly there appears to be quite a lot of singly ionized carbon generating 158 micron fine-structure lines -- ALMA also sees a fair amount of water, HCN, HCO+, SiO and a few other molecular lines at high redshift, and practically everyone thinks it's more reasonable to blame this on the earliest stars being enormous and dying young and very violently, rather than heavier-than-lithium nuclei fluctuating into existence far from what became (proto-)stars. The "Boltzmann brain" argument applies to much simpler things too, including carbon atoms and organic molecules).