You get it correctly. Deep learning isn't inherently good for cases where you need to be right 100% of the time. But we don't have a better tool so the industry is trying to see how far they can push it.

One big open question in self driving is: Do you have to be able to see/classify everything in the scene all the time to be a good driver? I think the answer is no (see: humans.) But, because most self driving stacks in the industry have chosen "3d classified representation of all of the objects in the scene" as a fundamental interface between perception and planning stages, there is a big challenge. (e.g. Representing uncertainty usefully is exceptionally difficult with this interface, among other things.)

I can imagine this being at least two orders of magnitude higher than most human drivers (one in ten thousand misses versus one in one hundred). How many orders of magnitude better does it need to be before it's good enough?

Do human drivers spot crosswalks in time with higher accuracy? That seems very unlikely to me.

Still, as long as the driver (human or otherwise) sees the pedestrian and vice versa, an accident can usually be avoided, with perhaps some angry shouting.