[EDIT] everything I could find with a quick search is about heavy, long-term use only, which evidently has little effect on white matter, and uneven effects on grey matter, increasing it in some areas and decreasing it in others. Nothing about long-term light use, occasional "binge" use, anything like that.
Personal experience is it's one of the few recreational drugs I can't handle as I experience extreme paranoia, rarely terrifying out of body experiences, and always an inability to speak, which is extremely uncomfortable as I deeply value my ability to communicate.
I get those too. After considering it for decades, I'm fairly convinced it's due to my psychological makeup - specifically fearing an unmasked loss of control.