Hacker News Reader

Comment by helltone | Hacker News Reader

One problem I encountered while working on robotics is that many commonly used algorithms yield approximate solutions with no error bounds. They work 99.99‰ of the time. This is fine from a computer science or math point of view, but very scary from an engineering perspective, specifically when there are humans nearby. A big part of me struggles to accept the suitability of algorithms coming from gaming engines or machine learning etc for real world heavy duty robots. The lack of rigour in the field is astonishing.

View on HN

This is the scariest part of using machine learning as an engineer on any practical application as well.

Without an error bound, ML can’t be in charge of anything that could put human lives at risk.

This is also why I don’t understand all the hype about FSD / L5 autonomous driving. We don’t even know yet if such error bounds even exist, so we don’t even know if machine learning is even the right tool for FSD yet. All certification entities for control systems that put human lives at risk in aviation, automotive, etc. require those error bounds. So it actually doesn’t really matter if Tesla comes up with a “maybe L5” system, without right error bounds, their cars won’t be certified as L5 and drivers will need to keep hands on the steering wheel.

What do you think the error bounds are for a human?

I know it sounds like a flippant question, but for certain applications, if we can get a model that's better than human, then it doesn't need to be perfect.

And they way we currently do this in all sorts of ways is to pair a human with a computer so that they each do what they're best at. It doesn't have to be about full automation.

Human is like your ancient software that was here since forever and somewhat worked fine. So everyone is used to it.

(the difference from an actual software is that humans are based on some crazy nanotech from the future that nobody can completely control)

> humans are based on some crazy nanotech from the future that nobody can completely control

Excellent summary :D

This is another one of that "extreme tail risk" scenarios, like climate change and GMOs, that people have wildly different and contradictory reactions to.

Sure, the "legacy" intelligence / climate / food could also have extreme tail risks, it's just that it's been tested for 100s of millenia... whereas new technology might be better in the average case or even 99th percentile, but the 1% (or 0.0001%) is unknown and potentially much worse.

However, it seems to me that people resolve this more along ideological / political lines than with any kind of rational reasoning.

> "extreme tail risk" scenarios, like climate change

Climate change isn't a "tail risk". It is a hard wall our civilization is approaching fast. If we do not solve it, it will undo the conditions we depend on to live.

What about "practical error bounds", i.e. testing the system through millions/billions of miles driven?

Miles driven is a useless metric. Stick your vehicle on a treadmill and have it drive a billion miles. What does that tell you?

Edit to add: a better metric would be something like "billions of decisions made where human life was at stake".

It's pretty clear the they meant "distance driven in ordinary conditions where humans normally drive cars".

That isn't the case though. Gm and Google both tell you that their incidents per mile (I think they use 100000 miles or something) is constant, because as they get better they test in harder situations. A automatic car on a desert freeway (no traffic) is easy compared to freezing rain during the afternoon rush hour in Minneapolis. (afternoon implying most people were out and so the decision to stay home isn't available). That is just on hard situation that comes to mind, I think those doing self driving cars know of others.

Empirical evidence has shown that all Tesla cars have the same failure mode when a stationary obstacle is on the highway. This is not some fringe failure that happens because of wrong classification, it's how the Tesla Autopilot was designed to function. Tesla has claimed explicitly that it's the driver's responsibility to avoid such a situation.

Tesla is not a good representative of the field; they are waaaay behind Waymo (Google) and Cruise (GM).

> testing the system through millions/billions of miles driven?

Makes sense doesn't it?

Well, actually no. The Google cars drive the same route every day in Mountain View with no deviation, so those millions of miles are really the same 10 miles over and over. Even seen 3 in a row behind each other.

Fools the regulators, and apparently you, though.

Deaths are often measured in deaths per 100,000,000 miles driven, with a current measurement of 1.18 deaths per 100 million miles[1]. If a car is driving 60 mph 24/7, it takes 190 years for it to drive 100,000,000 miles. Let's assume that, in a decade, 1% of cars are on autopilot. There are 37,000 deaths per year in the USA due to car accidents[1]. Therefore, in order for Tesla/Google/GM to claim that it is equivalent to normal cars, it has to drive 370 * (100 million/1.18) ~= 31,300,000,000 miles, with only 370 accidents. If $COMPANY really had, say, 400 accidents, then it would have to make up the additional miles. In specific, it would have to make up 30 * (100 million/1.18) ~= 2,500,000,000. That means a fleet of (190 * 25) = 4,750 cars driving 24/7 for an entire year. That's not impossible for $COMPANY to do, but it is starting to become unreasonable. What's more likely is that $COMPANY just straight up lies about its incidence rate to regulators. But that's hard to do, considering it's going to be public if someone gets hit by a car.

I don't think its possible to fool the regulators, but even if it was possible there are dramatic consequences for lying. Volkswagon tricked regulators into thinking their car didn't emit as much as it really did in 2007-2015, and ended up having to pay 33.3 billion dollars in fines and returns[2]. Volkswagon stock went from a high of around 27 to 12. 5 years later, it's only recovered to around 16. The massive lawsuit that would be formed if Tesla's (or other automated car maker) car was found to be unsafe would likely be even larger. Furthermore, it would crush the self-driving car business semi-permanently. VW at least still gets to make environmentally safe cars, Tesla and Waymo's main value proposition is dead in the water.

Basically, even though there are incentives to lie and cheat, I think the incentives to try to make something that is safe is far, far greater. And while I'm sure there are Google cars driving the same route every day, I'm sure there are also tests being done in all sorts of conditions all over the planet. I think that practically, socially, and politically speaking, we can use billions of miles driven. The stakes are way too high to lie.

[1] https://en.wikipedia.org/wiki/List_of_self-driving_car_fatal... [2] https://en.wikipedia.org/wiki/Volkswagen_emissions_scandal

This is how some hardware is already certified for use in safety critical systems. ISO 26262 calls it 'proven in use'.

> Without an error bound, ML can’t be in charge of anything that could put human lives at risk

Humans don't have "error bounds" either, and you trust them just fine.

I personally do not trust humans.

However, we do have inertia with humans running such situations, and until there is something provably/demonstrably better, I don't see the current situation changing.

Trust is a continuum. I don't see how you can say that you do not trust humans at least to some extent, no matter how tiny that sliver of trust is.

A good point. I do of course trust humans to an extent, but as a general rule I feel that it's been beaten into my head by enough experiences to expect for humans to mess something up.

This includes myself of course.

Humans are capable of generating and understanding creativity and complexity that are simply impossible for non-AGI automation. Even then, we don't just let people figure things out for themselves. We put them through training, and then test them. Even after that, we make them liable for negligence.

I don't think it's an obvious conclusion that error bounds aren't important for automation because they aren't calculable for a human. They are just very different beasts.

My point is not that it's not important. If someone comes up with a rigorous way to obtain error bars, that's great! I'll take it! My point is that trust should not and will not depend on it. How do you even quantify something like this to a layperson in order to persuade them?

Let's do a thought experiment: let's say, we had a self-driving car that's verifiably 10x better than human on average, yet does not provide "error bars". I know we don't have one now, and unlikely to have one in the foreseeable future, but bear with me here, for the sake of argument.

Would you trust it, rather than a random human Uber driver?

FWIW, I'm amazed that driving cars manually is perceived as normal every time I drive one. I can easily accelerate 2+ metric tons of metal to 100+ MPH, and get distracted, launching this deadly projectile with me inside into oncoming traffic. Most roads have _no dividers_. This does happen from time to time, lots of people die. Nobody gives a shit.

Humans suck so bad at so many things that robots will be better than them at a lot of fairly unconstrained tasks in the next decade or two. And I'm pretty certain they won't have error bars while doing what they do. Humans don't.

A human whose life is at stake controlling a car here at this moment vs a human who is far in space and time controlling someone else's car by programming it.

The other poster was talking about error bounds not error bars. That means they want to have a number that tells them how often a system fails and ideally a model of how and when it fails. So they want to have a system that comes with some guarantees. It works X% of the time, fails 100-X% of the time.

Note that X doesn't need to be high, or higher than a human, for example a self-driving car with a 60% failure rate is fine as long as we know when and how it fails. It's this "when and how" that's completely missing from self driving cars.

And in machine learning in general we just don't have any guarantees that a model trained on some dataset will perform the same way in the real world.

I think a lot of the nuance has been lost here. Even throwing around phrases like "10x better than a human" is already implying a lot of very vague measurement or knowledge. In what ways, and in what environments, is it better?

I've dipped my toes into robotics a few times, and every time I end up being reminded of just how painful it is. Even when you're working in an idealized simulator, it's extremely easy to find a little edge case that causes completely bizarre behavior. And moving out of the simulator only makes things far, far worse.

It's really easy to forget about those sorts of details and brush it away as just things to be solved while developing the automation. But they don't just go away so easily. Error bounds are there to help manage these issues and ensure we know how to best use the automation.

In many instances I think people would tend to prefer the Uber driver in a moderate reading of your scenario. A human driver is likely to perform somewhat consistently and predictably. If they drift around the corner and leave a long skid mark in front of your house you can make some assumptions about how they are going to drive. If you get in and see them struggling to keep their eyes open, you can again make some assumptions about their performance. A well-rested and safe driver is extremely unlikely to suddenly throw themselves into on-coming traffic with no warning. You can refuse or stop the ride if you judge you are not safe.

Automation is a different beast. It's liable to fail in ways that a human driver would not. It may be performing wonderfully until something a human would not even notice happens, and which point it may indeed throw you into on-going traffic. For example, look at adversarial examples in deep learning. As a passenger in this case you don't have a way to judge your own moment-to-moment safety. Even if it is safer on average, the sheer unpredictability and the resulting stress is likely going to eat significantly into any gains.

Humans have a mental model of humans, they can predict what another human is going to do with reasonable accuracy. Robots are wholly artificial so there is no existing mental model we can apply to them to understand why and how they act.

You have no mental model of the crazy person who drives his car into oncoming traffic.

Sure, and that's why we try not to let people with altered states of consciousness or with unpredictable rationality (children) drive.

Two counterpoints here:

1) Humans can estimate their own uncertainty. Ask a person to show how long a meter is, and they'll give you an estimate. Then ask them to show you the "error bounds", i.e. what they're "quite certain" the meter is longer than and shorter than. You are likely to get sensible bounds. Now, humans aren't amazing at this, but the brain does have capacity for estimating how uncertain it is.

2) No, you don't really trust humans. This very fact that humans are often imperfect in estimating their own uncertainty makes us very stupid sometimes. How many times you were sure you know something for a fact, for it to turn out to be completely false. This is why society tries to not put too much responsibility into the hands of a single person, or at least to provide help and/or safety mechanisms if that is the case.

Ask a programmer to estimate how long it'll take them to code something to see if we really have error bars on complex functions. The margin will be so wide as to be completely useless.

We do have error bars on simple measurements already. They're right there in the data sheet for the sensor. What you're asking for are error bars on things several levels distant in the layers of abstraction. Humans suck at that. We only cope the same way machines do: through constant negative feedback.

Programmers can give you good error bounds. Ask what the best and worst case is for solving a problem, and 80% of the time they will be within those bounds.

The problem is that management doesn't want to hear about the worst case, because it looks bad for them politically.

>> Humans don't have "error bounds" either, and you trust them just fine.

Yes, but humans are already here and doing all those dangerous and difficult things. But if we're going to replace them with something, it makes sense to want to replace them with something that's better than them. I mean us.

Otherwise, we might as well stick with the humans. Especially since we already know how to make humans (whereas self-driving cars, not so much).

I find the inverse surprising: that many algorithms that work on real-life robots _do_ _provide_ error bounds and their optimality / convergence properties are proven in the papers that introduce them.

A great example of this is motion planning, where papers both on sample-based methods (such as SST), and on search based (descendants of the A* family) argue at length the theoretical optimality and convergence properties.

On another note, I think requiring more theoretical analysis as a guarantee of safety could partially be an AI-winter meme rather than practical solution. Point in case: do people run a quick check of aerodynamics maths before boarding a flight? No - they rely mostly on the engineering and regulatory process that gradually made passenger flights safer.

It seems you are taking about theoretical error bounds, that is proofs in papers with assumptions on input probabilities etc. These don't always apply to actual implementations, in physical real robots. There is a huge gap in safety between practices in aerospace engineering and robotics.

Many localisation algorithms are probabilistic, eg. 'you are here within R of X'. A couple of times I've had a manager who could just not accept the probabilistic nature: "It's just right there!"

So I to explain my manager that we just cannot do better and know for sure that the robot is really in position X, especially with the limited sensing the project would afford. Sure, you can do the classic AGV thing and add magnetic markers everywhere. Or use more sensors to get higher accuracy, but none of those were popular options.

Why bother, you'll hit the uncertainty principle sooner or later anyway.

The solution generally employed is to separate humans and robots if the robots are moving at speeds or momentums that could harm a human. There are safety regulations to be complied with and the general idea is that your robot should not be able to harm a human no matter what error your software suffers from. This is sadly impossible with self driving cars but applies to most of robotics.

Afaik there is some progress being done on motion planning and control parts with regards to that, mostly using reachability analysis for continuous systems or using things like control barrier functions.

I work in the perception/localization domain and I am not aware of any large developments in that direction. I do know that there are certain ML based perception systems that got some levels of ASIL certification.

Purely in the space of geometry and simple motion planning, there are reasonable methods with guaranteed correct solutions, provided their sensor data is correct. Of course, combining correctness (the solution is actually safe), completeness (a solution will be found if it exists), and bounded computation time (get a solution in actually useful time) is a much harder combination to achieve.

Though, probably humans fumble or bump things >1/10k times. I certainly do, anyways.

Also, replace deer with something self driving. Those things are so stupid when near the road.

99.99% is pretty fine from the engineering point of view. The buildings and other constructions surrounds us have about the same theoretical reliability considering all the uncertainties involved like weather, load and impacts, long term characteristics, material and manufacturing uncertainties. Of course this centuries long trial and error supported pretty simple science of construction engineering needs to be supported by law level standards and regulations allowing certain low level uncertainty otherwise the 'engineers want to sleep at night' aspect was forcing us to make only very expensive bunkers to live in (or only sociopaths becoming engineers). There is an accepted level of risk involved in engineering.

Usually mathematics and (proper) algorithms is the topic where everything is 100% (good quality actually finished work, not an early prototype released as final is assumed). It just may or may not to be fully relevant to our life.

> The buildings and other constructions surrounds us have about the same theoretical reliability

You are off by a few nines.

Transportation machines are expected to have 5 or 6 of them. And those are the most dangerous kind we keep around. Everything else is more reliable.

No, I, am, not. It is about 99.998% Please do not cite numbers from different fields.

By that number you mean that 2 in 10000 buildings fail at some point due to some design issue? Because building reliability is a very hard concept to define, even more in the context of machine reliability (we are discussing robots here).

Anyway, machine reliability is calculated over usage, not lifetime. It gets much larger numbers.

Reply on news.ycombinator.com