Yeah I do believe it makes a difference, because brain atrophy is linked to a range of cognitive impairments, for example strongly correlated with Alzheimer's disease.

But you're right on the last point though, about assuming thinner is bad automatically. Heavy and chronic cannabis usage has been linked to an increase in density, but a reduction in total volume, which resulted in a slight overall decrease in IQ. The takeaway from all this being: Cognitive function is affected by a variety of parameters and we got more neuroscience to do still.