"It doesn't know how it knows what it knows" is a very common criticism. "It's just predicting the next token but doesn't really 'understand' anything" is another.

To me it's like saying, "How can humans actually play baseball if we don't even explain how we throw balls and swing bats at moving objects?" Wording it that way, it just sounds ridiculous. As ridiculous as I think this AI conversation is.

I don't know why we care so much about model transparency. Yes, it's worth scientific pursuit to understand what we're building and how it works, we should absolutely try to know as much as we can. But if the bar for "true intelligence" is a machine capable of doing things that we can't do ourselves, is that saying we aren't "truly" intelligent, either? If we're all of a sudden not allowed to leverage systems we don't fully understand then I guess we shouldn't even be at the spot where we are as a species. We've been doing the right things for the wrong reasons for ages, it seems to work OK.