I'm surprised he made the drug analogy. It takes decades and hundreds of millions dollars to be able to test a drug on humans. I'm curious if he thinks we need a similar safeguard for all AI-human interactions.
It takes that long and costs that much because there is no safe way to do this with human physiology other than to carefully, slowly and expensively try it and measure the consequences.
We can take any AI system and model its behaviour with statistical certainty much more cheaply and quickly, and be more confident in its future behaviour.
Remember, AI and ML are fancy words for some form of a regression to a mean more often than not. If we run hundreds of millions of tests of such a system in simulation for a very wide range of contexts/inputs (which is cheap to do), we can have a much higher degree of confidence in a short space of time around behaviour of that system than we will for any drug test, even if we still don't fully understand the causality.
But we are quickly moving beyond the types of applications where you can simply test in simulation. Self driving cars are probably right at the edge of this. It's what comes next that worries me. And I don't pretend to know what it will be.
So even when we do not understand the precise impact of drugs on humans and there is no safer mechanism to test, we leave only 0.5% of candidate molecules to empirical/statistical evidence in the form of clinical trials.
On the other hand if drug discovery was treated as a pure AI problem, we would have thousands of unverified and unsafe molecules in clinical trials.
Causal principles get us to 99.5% of the way in drug discovery. Unfortunately not so in AI.
You're still left with double-blind trials and having to get large sample groups to try those molecules though.
And it's for that reason that drug discovery is always likely to be quite slow, complex and expensive - the efficiency gains will be pushed towards the top of the funnel to make new ideas reasonable to explore, I would imagine.
My point was that when you're not dealing with human physiology and instead dealing with problems that are more tractable through AI - i.e. using regression to tune algorithms through patterns in data - you are going to get quicker and more impactful returns without the same complexity.
And - critically - it's OK to often trust the AI solution you have without understanding causality. If you later find it's doing something odd that is undesirable, you can use that data to help tune the algorithm again without having to understand the causal relationship.
Put another way, you can teach an AI to get better without necessarily understanding the subject completely yourself.
Finding something odd for an algorithm (especially a deep neural network) is hard because they fail in just so many ways. For example, lenet for mnist almost always gives high confidence predictions for random tensors(torch.randn). Most imagenet models fail in the presence of just 20-30% salt and pepper noise. (Both of these are problems solvable through simple preprocessing techniques)