Hacker News Reader

Comment by shafoshaf | Hacker News Reader

The only barrier for higher stakes applications is going to be the frequency of errors. Flying an airplane or running a factory has a lot less margin for error, but humans don't do those things perfectly either (Chernobyl, Three Mile Island, Union Carbide-Bhopal disaster). It doesn't have to be perfect, just better than humans. And in fact, I'd argue that by having no deterministic outcomes prevents systemic failure, like having a single point of failure for all those drones in the crappy episodes of Star Wars.

View on HN

> The only barrier for higher stakes applications is going to be the frequency of errors.

Frequency and strength. My issue with e.g. image classifiers is that when they’re wrong, they’re catastrophically wrong — they don’t misidentify a housecat as a puma, they misidentify a cat as an ostrich.

Or, put in the language of risk, probability and severity

The only barrier for higher stakes applications is going to be the frequency of errors.

IE, "The only barrier to the software working perfectly is it's tendency to fail".

Which is to say this sort of argument effectively assumes, without proof, that are no structural barriers to improving neural network performance in the real world. The thing is, the slow progress on self-driving cars shows that reducing the "frequency of errors" can turn from a simple exercise in optimizing and pumping in more data to a decades long debug process.

>It doesn't have to be perfect, just better than humans.

I have a different opinion on this.

Humans don’t like uncertainty. We like to feel like our mental model of reality can predict future outcomes. When it doesn’t, we get very uneasy. It’s why we don’t like dealing with erratic humans.

Part of the problem with AI is it’s lack of interpretability. People aren’t going to want to interact with AI if they can’t intuit what it will do, even if you can show it’s statistically better. The performance barrier is going to be much higher than just a little better than humans. We don’t have that limitation when dealing with people because we can more easily infer their goals and actions.

Thinking that being a little better than humans is the threshold is a rational decision. But human trust is often irrational. The latter often drives politics which can regulate AI into a corner.

Better than humans is not really meaningful as human skills have a very wide range and they can even vary depending on the circumstances and available resources.

Also, on average is not a great target either, sometimes it makes sense, but there are plenty of examples where we definitely don't want more average work.

But there's the rub. It's impossible to determine through testing whether a particular AI system will actually have a lower frequency of errors than humans. You can program an AI system to handle certain failure modes and test for those in simulation. But complex systems tend to have hidden failure modes which no one ever anticipated, so by definition it's impossible to test how the AI will handle those. Whereas an experienced human can often determine the correct course of action based on first principles.

For example, see US Airways Flight 1549. Airbus had never tested a double engine failure in those exact circumstances so the flight crew disregarded some steps in the written checklist and improvised a new procedure. Would an AI have handled the emergency as well? Doubtful.

If you understand the problem well enough that you know that the frequency of errors is stable and won’t blow up on out-of-sample data, then it seems like you’ve fully modeled what’s happening? That’s not any easier than solving the problem because it is solving the problem.

If don’t have good reasons to be confident that the error rate is stable, then you’re just guessing that you solved the problem, because it seems to work.

The problem is that humans fail in very different ways. I don't ever need to wonder if a human driver following behind a road maintenance truck visibly carrying traffic cones or stop signs is going to be a problem...

There's a weird bit of induced demand like widening a freeway, makes errors less often than a human but is more scalable so the absolute number of errors increases. I guess in the case of self driving cars, it could be from hordes of autonomous shipping trucks that outnumber existing truck drivers.

Reply on news.ycombinator.com