On the other hand, we'd like to know how it works. I mean, why do we do science at all? If you don't need to know physics to throw a ball, why bother studying mechanics or biology?
To me it's like saying, "How can humans actually play baseball if we don't even explain how we throw balls and swing bats at moving objects?" Wording it that way, it just sounds ridiculous. As ridiculous as I think this AI conversation is.
I don't know why we care so much about model transparency. Yes, it's worth scientific pursuit to understand what we're building and how it works, we should absolutely try to know as much as we can. But if the bar for "true intelligence" is a machine capable of doing things that we can't do ourselves, is that saying we aren't "truly" intelligent, either? If we're all of a sudden not allowed to leverage systems we don't fully understand then I guess we shouldn't even be at the spot where we are as a species. We've been doing the right things for the wrong reasons for ages, it seems to work OK.
What happens then ? If it’d like to do harm for any reason (it might not be harm for an AI but just the « right thing » according to its criterias) what would we do ?
Wouldn’t it be a good thing to know how it works inside precisely ? Or « whatever happens, happens » ?
That could refer to "knowing the physiology of my brain", which is what you seemed to be referring to. But it could also refer to "explaining my feelings or decision making process". I've lived in my head for many years and had many chances to observe myself, so I can perform post hoc self-analysis with reasonable accuracy, and on a good day, I can even be self-aware in the very moment (which is useful in getting myself to do what I want).
I think maybe that second thing is what people are worried about AI lacking.
> If we're all of a sudden not allowed to leverage systems we don't fully understand then I guess we shouldn't even be at the spot where we are as a species. We've been doing the right things for the wrong reasons for ages, it seems to work OK.
I don't say this merely to be contentious, but I don't think I have quite as optimistic an outlook myself ^_^' which isn't to say I think we shouldn't meddle with the world, just that we sometimes get ourselves in over our heads. But I'm hopeful.