I think part of it for me is that I do not have an intuitive understanding of the box model and how layouts are calculated. On top of that, since CSS cascades, you have to be able to mentally keep track of all the parent attributes to understand how they affect a specific element on the page.
I was working on a React Native project the other day and I could just not get flexbox layout to do what I wanted. I ended up just playing around with putting attributes on various components until I found what worked.
I have an ADHD brain that quickly discards information I don't use frequently, so since I don't use CSS often, I never remember it well enough to be good at it.