That could refer to "knowing the physiology of my brain", which is what you seemed to be referring to. But it could also refer to "explaining my feelings or decision making process". I've lived in my head for many years and had many chances to observe myself, so I can perform post hoc self-analysis with reasonable accuracy, and on a good day, I can even be self-aware in the very moment (which is useful in getting myself to do what I want).
I think maybe that second thing is what people are worried about AI lacking.
> If we're all of a sudden not allowed to leverage systems we don't fully understand then I guess we shouldn't even be at the spot where we are as a species. We've been doing the right things for the wrong reasons for ages, it seems to work OK.
I don't say this merely to be contentious, but I don't think I have quite as optimistic an outlook myself ^_^' which isn't to say I think we shouldn't meddle with the world, just that we sometimes get ourselves in over our heads. But I'm hopeful.