It's a big deal as far as impact, sure, but I think it's also OK for us to abandon the need for complete control. Because, really, that's what this is about: we're worried that if we can't understand what's happening, we can't control it. Personally I think that's just fine. It's a different class of thing but that's also OK.

Do we even really know all that we should about how to manufacture chips? My understanding is that we rely heavily on experiments and heuristics. I think with complexity that's just how things are, sometimes.

And again, going to the human-throwing-a-ball metaphor, maybe generalized intelligence actually requires introspective opaqueness. Maybe it's some cosmic law we don't understand: the better a system is at handling open-ended issues the less it can be systematically understood. I just think the utility of application is so, so much more important than our inability to know down to the last bit how a given LLM works. If it works, and delivers value, then we just don't need the why. We can and should try to understand, but we can also accept that we won't always get all of the answers we want.

I mean, why does anyone do anything? Ask someone to explain why they did everything they did today and I'm sure a lot of what they'll tell you is made up or just plain wrong. Humanity seems to be just fine despite all that. Why do we expect our apprentices to be different?