“It's a truck full of traffic lights”

It's fascinating to watch this technology develop, but as a driver I usually feel more anxious using "autopilot" that not using it.

With one huge exception: self parallel-parking.

I don't understand why this innovation doesn't get more love. Tesla's not the only manufacturer to offer this, of course, but this particular innovation has increased m enjoyment of city driving more than ... well, more than anything I can think of.

Even if they never get anything else to work, it would have been worth it just for self parallel-parking. lol

From an engineering perspective, of course there are serious problems that need attention, but sometimes it also good to celebrate the wins.

That's basically true, but there's no reason to limit it to Autopilot. Although the high points of AP are better than any other system that I've tried so far. I've seen it do amazingly well in the rain at night, and in the rain on a road with faded markings that were hard to pick out. I've also seen it thoroughly confused by double-dashed lines at hov lanes.

I'll still take that over the ones that beep proudly to tell you that they are about to fail to maintain their lane (hi ProPilot).

The first thing that crossed my mind when I saw the clip of autopilot hallucinating flying traffic lights was that it was clearly DUI and there’s no way I’m getting in the car with that guy.

I actually don’t want autopilot for the fast, freeway speed driving. That’s easy. I want it for the mundane, exhausting stop-and-go, bumper to bumper traffic as we all slowly creep through the traffic.

That’s better than dead grandma who is leaning her foot on the gas just so to get you to keep the same speed aka cruise control. Dead grams is in many many cars.

Disclaimer: I work on the business side of the autonomous team at a major OEM.

The autonomous car business model is very much alive and well, it has simply shifted in two key areas that cause general consumers to misinterpret what is happening.

1) Use cases (ODD). Many people think "if I can't buy an autonomous car, it's not a real thing". But most manufacturers and AV developers other than Tesla aren't even trying to tackle that use case. Instead, the focus is on autonomous vehicles operating very profitably in specific locations and operating models that work. Heavy trucking on "easy" routes, ridesharing in well-mapped areas of the Southwest US, shuttles on popular metro and airport routes, etc. These use cases represent very large revenue opportunities and the tech development is progressing well.

2) Timing. Too many pundits and CEOs looking to generate PR buzz made ridiculous claims over the last few years about when autonomous vehicles would be ready. This set false expectations among the general public which has now soured perceptions since autonomous tech is not easy. Fully driverless is already ready in some very specific scenarios (ex. Waymo in Chandler) and is on track to roll out to more ODDs over the next 5-7 years. I can pretty much guarantee that you will see a significant number of fully driverless vehicles in suitable areas by 2030. No, they won't drive in upstate New York in a blizzard, but no one actually cares about that from a business perspective.

The other issues you raise - liability, insurance, legal roadblocks, etc. are mostly non-issues that already have solutions. The only one that's a continuing problem is the fragmented nature of legislation across different states. But there is very heavy lobbying going on right now to rectify this by the time it's actually needed.

> My conjecture is we end up building AV only roads. initially one lane of a highway, then ring roads round cities and major warehousing hubs, then across urban areas. Walled off in some way they simply become railways with benefits.

Close, but I think it's far more probable that we just whitelist roads / intersections / routes known to work well and be reliable--whether with a general-purpose or location-specific algorithm. There's still a ton of value shuttling people to and from the airport and local hotels.

Then there'll be incentive for local restaurants to also be included in the available destinations, so there'll be pressure to both improve local roads / intersections to make them more easily navigable by self-driving cars, and on self-driving cars to get better at navigating them. With time, the locations / routes that self-driving cars can reliably reach will expand until it covers the vast majority of desired destinations.

We actually are following the same pattern with the horses -> cars transition. There are still plenty of places that cars can't reach and horses can. Over decades, though, we just kind of paved roads everywhere anyone really wanted to go. Nowadays it's more or less a given that if you want people to be able to reach somewhere, it'll need to have a road.

We'll see the same thing with self-driving cars. There'll still be roads that self-driving cars avoid and humans drive on, but they'll become less and less relevant as the value of self-driving cars becomes more apparent.

It doesn't need to be almost perfect. Waymo, for instance has remote human assistance when it gets stuck, and even a human in a normal car drive to the scene and to get it unstuck if it's really bad. There could well be a point where those humans plus the tech are cheaper than a traditional full-time-per-customer human taxi driver.

You might say people don't want those delays but we tolerate delays in normal commuting. An unattended bag on a tube station stopping trains, a car accident blocking traffic, traffic jams blocking traffic, mechanical breakdowns, etc. As long as they're infrequent enough, it should be OK for riders.

I see this rather differently. In fact AI is already as good as human drivers, statistically. Only a few years back, we could point at people getting killed by software bugs like the Tesla path-under-semi-trailer issue (not a lot, but it did happen and they were real bugs).

That's not happening any more. All we have left is laughing at stuff like this, where the visualization (not even the autopilot!) gets confused by seeing real (!) traffic lights on a truck, so it paints them in space, but then has to re-recognize them because they are moving.

At some point, the luddites will just run out of ammunition. It's sort of happening already.

I still have hope, because we have two different mentalities at play.

The individual still thinks in terms of "I'm an above-average driver, I'm better than a silly bleep-bloop box." They'll resist self-driving en masse if it's not 1000% perfect.

Insurers, however, will see self-drivers as predictable, never drunk, always following the speed limit, and in aggregate preferrable to humans. They will be able to see "3% better" as meaningful even if it's not perfect, and price accordingly. Eventually, you'll pay a rapidly increasing penalty rate for having a steering wheel. This is where the avalanche comes from.

I also figure self-driving could unlock new features-- in particular I'm imagining the only way we get road speeds much over 120kph is self-driving, as the human response time becomes a limiting factor. And of course the in-dashboard wet bar. This will feed back into the consumer market even if it's only closed markets at first: why can't I go 250kph on the highways considering the self-driving airport shuttle does?

I have been saying for years that full self driving is extremely unlikely to be developed any time in the near future because it is simply too hard of a problem for any reasonable near-future extension of currently available technology. I am gratified to see that the conventional wisdom has recently come to also see things that way. I am not surprised that journalists bought the hype - I assume that half of them have no idea what they are writing about and/or are just advertisers in disguise. But it does surprise me that so many experienced corporate professionals and investors got taken in. I guess maybe part of it is that they just are not as smart as I once thought they were and maybe another part of it is that some of them used the hype to grab a bunch of cash with both hands, cash that the people who put it up will now never see again.

I think the issue is bigger than just self-driving. AI/ML is wildly oversold as an engineering solution to real world problems. In the best case, it's solving easy problems under narrowly defined conditions. It's not really solving hard problems robustly under real conditions.

You'll know self driving is almost here when it works in simulation at non-interactive framerates.

Au contraire, seems we're mostly there. It's incredible having watched self-driving, and neural networks, go from new concepts and barely functioning to "order here" and public use (tell me the numerous YouTube FSD videos aren't what they are). Insofar as there are still serious edge cases to address, they're being solved.

Time and again I've watched "ain't happening" technology become the preferred norm practically overnight. Eagerly awaiting my FSD CT, and making long trips without having to micro-manage every foot across thousands of miles.

you can skirt all regulation and set false expectations by simply calling your driver assist feature “autopilot”

it is very unfortunate that the first & really good electric vehicle is a highly complicated tech gadget. and this sets a bad benchmark for the other companies which want to step into this space. why not a simple electric vehicle : just different fuel and engine , and all the rest same as ICE vehicles . wouldn't that be easier / simpler to deal with ?

sadly, in India too, the major electric 2wheeler Ather, is kinda going the same high-tech way as Tesla. and very expensive !

Nope. You're just not patient enough. ACs will come, but very gradually. They're already (mostly) here but it will take another decade or two to be completely A.

> My conjecture is we end up building AV only roads. initially one lane of a highway, then ring roads round cities and major warehousing hubs, then across urban areas. Walled off in some way they simply become railways with benefits.

That would be awesome still and make people want to have that technology...how does that murder Tesla's business model? Waymo would easily pivot to licensing their tech to car manufacturers.

I don't need autopilot until it is 100% (or 99.999%) but I'll be damned is cruise control "auto pilot" in a stop and go commute is nice and works well enough. that takes a bit of the grind off the drive to work.

Railways with benefits aren't good enough and would be an even worse business model.

Doesn't "I'm fairly confident" kind of imply you're not actually confident?

I'm curious what the issue is here that seems unsolved? It's unconventionally displaying what it is recognizing, but the car isn't doing anything janky.

The car is properly recognizing traffic lights pretty darned well, considering the circumstance. It looks like it has a built in understanding that traffic lights are "always" stationary - hence, assigning them static locations on the 3D map - but it keeps having to update the model because the lights are actually moving.

This seems like a very non-obvious edge case that I wouldn't expect an ML team to even consider as a possibility. Now they need to program into the ML model an understanding that traffic lights are typically stationary. Which seems even more difficult to me, from a technical perspective - you don't want false negatives...

The car isn't braking or making any strange maneuvers from what I can tell. I'm actually impressed that it's handling it this well.

Years ago, a friend worked in the Autopilot group. It took them a year to procure servers to store the telemetry of the existing cars, and then weeks to have them setup.

They don't work there anymore.

From their experience, I know one thing: I will never work for Elon Musk. He may be a great visionary and salesman, but he's a horrible manager.

> I think the real problem is obvious.

Yes. The human brain and visual systems aren't nearly so trivial to replicate as a lot of people in the tech industry seem to think.

Tesla is just one of many case studies in the paired tech industry arrogance seen so frequently:

- "A human is just a couple really crappy cameras and a neural network, we know how to do better cameras and neural networks, how hard can it be?"

- "We can do anything we dream with 99.995% reliability in the synthetic, computer-based world of the internet because we know code. Therefore, we can do anything we want in the physical reality with code!"

Both are far from evident in practice, but the belief in them continues, despite it being increasingly obvious to everyone else that neither one is true.

Human vision and world processing is quite impressive - and, as pointed out elsewhere in this thread, a two or three year old would have no trouble working out that the obstacles were some things on a truck. I've got a nearly three year old, and I guarantee he wouldn't confuse those for stoplights in the slightest. I also wouldn't let him out on the road, though he does well enough with a little Power Wheels type toy. But there is far more going on in the visual processing system than we even understand yet, much less have the slightest clue how to replicate.

And while code may be fine on the internet (where you can retry failed API calls and things mostly make sense), the quote about how fiction is constrained by what's believable and reality sees no such restrictions is very true. Out on the roads, all sorts of absolutely insane things can and do happen on a regular basis - and you can't predict or plan for all of them. But the car has to handle them or it crashes.

As a random example, a year or two ago, I was behind a car that had poorly strapped a chunk of plywood to their roofrack with a good chunk hanging forward, and the front end of it was starting to oscillate awfully hard. I had a good clue that it was going to come apart sometime in the very near future, so backed off from a normal following distance to quite a way back. Sure enough, half a mile later, it failed, went flying through the air, slammed into the road a good distance behind the car, and tumbled a bit. Had I been using a normal in town following distance, it would have either hit me or tumbled into me, but using a human visual system, it was obvious that my existing following distance stood a good chance of being a bad idea.

Stuff like this happens on roads constantly. Meanwhile, state of the art self driving can't tell the difference between stoplights and some poles on a truck. You'll excuse me if I don't think the problem is anywhere remotely close to solved for a general case Level 4 purpose.

I've never understood this argument. Isn't the main bottleneck that you need well-labeled data for training the neural networks? How is having tons and tons of random camera footage going to help?

As another ML engineer I have all the respect for Andrej Karpathy and wouldn't dare to walk in his shoes. My ML model could kill people? No thanks, I couldn't sleep anymore.

Right? This should be trivially solved with some sort of temporal filtering on detected objects. If you detect a traffic light at (x,y) in one frame, and it disappears in the next, but there's a new one at (x+dx,y+dy), then you shouldn't place a new one down in the world frame. You should only place a traffic light down if you're confident it exists and is operational. At the very least, the lights should be detected on the back of the truck, but they should move with the truck. At least that matches what's happening.

I don't understand why this is hard for Tesla engineers -- I was doing this kind of thing in grad school a decade+ ago and it worked fine. I've seen it in other demos where object classifications rapidly cycle between person, bike, car, etc. Are they not filtering anything? Is this a symptom of "AI-ing all the things"? Because we did it with bog standard computer vision techniques back then and never got behaviors like this.

> they should have a million hard test cases that must pass

Move fast and break things [like tests]

I saw the video and it is crazy that they think it is even close to production.

When I got my Tesla back in November 2019, when stopped at traffic lights, cars next to me would be shown to rotate randomly and jiggle and dance.

It works much better now, but it was always kind of amusing. It only did it when we were all stopped. Moving cars (whether it was them moving or me) were rendered just fine. My guess is that it was extrapolating movement based on more than one "frame" of sensor data, but at a stop there's no changes beyond noise, and the noise was being extrapolated into extreme movements.

I have seen it show cars "crashing" into me, but it's only semis when in the neighboring lane.

Not if the GDPR has anything to say about it

> The data subject shall have the right not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significantly affects him or her.

https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CEL...

We still use electricity even though it's deadly and it accidentally kills people every year. Why not use ML even if it has it's own issues? As humans who know better we're responsible to take all the precautions.

I bet carrying streetlights in a truck exposed like that is going to become illegal. Also painting bikes on trucks.

Knowing what i know about ML, it's the problem if we are rellying on cars to see unusual cases on the streets for the first time and handle them well.

That’s what’s frightening about this. I feel not enough people internalize just how primitive the car’s understanding/perception of the world is. A two year old human brain would have no confusion in this scenario.

We discount how much the concept of “understanding” is required in visual perception. We don’t just see shapes we have a complete understanding of what we’re seeing.

I might see a big rectangle flopping in the lane in front of me. I can immediately ascertain whether it’s a piece of tumbling plywood, or foam based on movement characteristics, color, apparently size, etc. I can then use that understanding to decide what evasive actions are required.

A Tesla it seems has absolutely zero of this capability.

And that illustrates the important problem with self-driving car: if you want L5 autonomy, you need to be handle all the weird cases.

who isn’t using that strategy? openpilot uses it.

A demonstration of why it will be a long time before it'll be rendered safe enough to drive on its own

The car is not under autopilot, the driver is driving. The car is just displaying what it thinks it can see.

(the grey steering wheel icon means autopilot is not engaged, it would be blue if it was on)

At least in the clip it's accelerating, so it doesn't seem to take them into account - which would make sense for non-lit traffic lights anyways?

I don't think autopilot is actually enabled in this clip, the steering wheel icon is greyed out.

I mean it makes sense. If you've never seen traffic lights be delivered to a new intersection you'd probably blithely assume you'll never encounter a stack of them that are unlit and moving.

Somewhere at Tesla there's a junior engineer who's telling a senior engineer "I told you so!"

> The odd thing is that even a randomly stupid AI for self driving is statistically safer than most drivers.

A strong claim, lacking actual evidence for it. All we have to go on are some Elon tweets (rather the definition of a biased source) and the actual crash rate. Without a lot more data (which Tesla steadfastly refuses to release) about environments, corrections, etc, it's quite impossible to make that sort of statement with any confidence.

The Tesla hardware is a weird combination of capable and insanely dumb, and it's far from obvious which it will be in any given situation until it's gone through it.

If an honest statistical analysis of the data indicated that Tesla's automation was better than human drivers (or better than other driver assist systems), I would fully expect them to have released the values. Since they haven't, and only hint at it and make statements that sound statistical but really aren't, I assume they've done the numbers internally and know it's not nearly as good as they like to imply.

If I drove in a city like their "self driving" beta was a few months back, I would be hauled from the car on suspicions of driving while hammered.

> The odd thing is that even a randomly stupid AI for self driving is statistically safer than most drivers.

This hasn’t been shown yet at all. Statistics showing autopilot have less crashes per mile always ignore that Autopilot is doing the type of driving that has the least accidents per mile (motorway driving).

> even a randomly stupid AI ... Is statistically safer than most drivers

the fact that people still trot this out every Tesla thread is super annoying. Sorry to break it to you but this is a 100% false claim

I was wrong, it's almost as safe, not safer.

https://www.forbes.com/sites/bradtempleton/2020/10/28/new-te...

Why hire a team to beta test your AI self driving car when you have customers willing to shell out $10k for the privilege?

> The odd thing is that even a randomly stupid AI for self driving is statistically safer than most drivers. Under sunny highway conditions? Perhaps. In a night snowstorm? Probably yes - because the AI would be at the side of the road waiting for the human to drive.

I think self driving is a typical 80/20 problem. We won't have "full" self driving because the costs are exponential for each step closer to it. But driving on 80% of roads on 80% of days, with supervision? That could happen.

But that said: we won't accept AI that just makes traffic safer "on average". I'm fine with human shortcomings causing accidents. People will not accept car manufacturers cutting corners and causing accidents, even if statistically it's safer. So the very high bar for self driving isn't just "as safe as humans".

> Red Team

I'm sorry, did you mean customers?