Tesla stopped reporting its Autopilot safety numbers online. Why? | Hacker News Reader

Tesla stopped reporting its Autopilot safety numbers online. Why?

latimes.com

108 pointsgamblor956151 comments

As a robotics engineer (though I don’t work on AI systems) I was bullish on Tesla’s architecture for a long time. I’ve watched all their presentations. Their hydra network. Their massive custom AI chip. I really thought they had a chance at making it all work. But the long tail is killing them. They’re walking back their claims for fully autonomous operation. Andrej Karpathy (their AI lead) left. Musk is distracted and showing that he doesn’t have all the technical chops we thought he did (if reports from inside twitter are to be believed). It feels like the company is all promises and no delivery now. Many have been saying this for years, but I really thought they were going to pull it off. Now I’m beginning to feel that they’re not organized appropriately to do it. You just can’t have a single guy who is all ego running a project as complex as this and find success. I think of how they must be doing things at Waymo or Cruise and they’re going to have a team of competent people in charge who don’t make rash decisions on major architectural issues. And given the scope of this problem I’m really starting to believe that’s what’s needed to make this work. There’s just no silver bullet. I’m really questioning what led me to believe in all this in the first place.

Tesla's approach has always been the least credible of the self-driving companies: they seemed to belive that if you just iterate on driver assist enough you get self-driving, and they seemed to want to make the system work on far more constrained hardware than their competition (because they were for some reason aiming at selling direct to consumers, which is not a particularly good business strategy for self-driving. A cynic would suggest it was vaporware for marketing all along). Other groups working on the problem (especially Waymo) seemed to actually grok that this is by far the most complex safety-critical system ever envisioned and approached the problem accordingly (with a lot of caution, laying of groundwork, a plan to roll out the system gradually, and bulding the it from the ground up to work on its own, not as a driver assist).

When Tesla got rid of radar and switched to "vision only" I cynically assumed it was a way to compensate for supply chain issues rather than radar being as useless as Musk was saying. I still think this is a possibility.

It is certainly possible to make a self-driving system with vision alone.

But it is also extremely likely that this is much harder than doing it with radar and lidar added to vision. Since nobody knows how to do any of them yet, you can't expect the company trying to solve the much harder problem to be one of the first to get it.

Their communication is a constant denial of this fact. So they either have no idea what they are doing, or are lying through their teeth.

It's also possible their radar was functionally useless: while it gave good range information it had poor angular resolution, a fact which seemed to contribute to Autopilot's issue with recognising stationary obstacles. it's worth keeping in mind that LIDAR gives much higher quality data than low-cost radar. (It also doesn't follow that if their radar was useless that vision alone would be good enough)

That might have made more sense to me if they hadn't followed it up by removing their ultrasonic sensors, which were functionally quite useful, since without them the car cannot see things at low speeds that are below the plane visible from the front-facing camera mounted opposite the rear view mirror.

Except cameras don't work in low-vis situations. Their range is shorter.

In fact, when Tesla removed the radar their max-speed with self-driving (or what is it called) dropped from 80mph to 75mph.

What's worse is that they disabled the radar for everybody. How is that not a class action? You're losing features.

I think it sounds like Musk looked at humans and went "we do fine with just vision, so make the computer do it with just vision, that has to be good enough"

Which underappreciates the fact that it is such a hard problem (nature had 500M years to iterate on the reference design) that you want to cheat as hard as possible by giving yourself the best sensor information you can.

Automatic cars need to be better drivers than us, not just good enough, as another thing.

If they were as good as the average sober and undistracted human they'd probably be a massive improvement.

But yeah, they need to hit the highest bar possible. As we've seen with vaccines anything developed by humans has to be nearly perfect, while a "natural" massive clusterfuck with 40,000 deaths per year won't count for anything by comparison.

> If they were as good as the average sober and undistracted human they'd probably be a massive improvement.

How do you convince the better half of drivers that they should use a system that is worse than themselves, personally?

If you had a system that drove roughly as well as I do at my 90% best and cut out the worst 10% of my driving, and let me not have to deal with the road that'd be fine for me.

And I probably should have said "sober, undistracted and competent human" since some people just shouldn't drive, and some people shouldn't be driving under certain conditions (which may not be tired or drunk, but going through a rough break up with late night emotional blowups).

The argument from Andrej Karpathy is that humans use vision and vision alone, so if we can learn to drive by vision, then so should an ML model. A radar is a sensor a human doesn't have, but we can gauge distance, and they also do introduce potential supply chain issues. So limiting the number of sensors is beneficial there. He also makes the argument that you have a tonne of data already. 8 1.2mp cameras, so 9.6 million pixels or effectively 9.6 million separate data points used to train the model. When you have that much data, what does including some relatively low resolution radar information give you that you couldn't infer from vision?

I don't know enough about AI to know if he's right, but it makes some sense to me. I think my biggest issue with it is that human eyes are insanely high quality, able to adapt to huge changes in light levels and clean themselves every few seconds. I know Teslas have issues with cameras being blocked by sunlight and things like that. A human can just move their head and use the sun visor.

With my limited knowledge I think vision probably is the ultimate way forward, but maybe they're too early. Perhaps there's some breakthrough we need to simplify the problem. I don't think Tesla will be the first to start making money from autonomous vehicles. I think Waymo will be the first. LIDAR isn't scalable but it simplifies a big part of the problem to the point that you can generate revenue from autonomous taxis while you train and perfect the general purpose vision model in the long term.

It doesn't make sense to me at all.

We have an iris, this allows us to limit the amount of light hitting our retina, or increase it when conditions are darker. Do the cameras on a Tesla have shutters that can do this? Otherwise you're going to have fun when there is glare off a wet road.

Did you even read my comment?

The issue is that self driving needs to be better than human drivers by a considerable margin. The public will not tolerate an AI driver that makes the same mistakes as humans, they expect better. To do this it will need access to sensors that humans don't have

A good human driver won't cause an accident though. The majority of accidents are caused through distraction or inhibition. A car can't get distracted or drunk.

If you limit yourself to the equipment that humans have, how do you expect to outperform humans? Would we accept 40k deaths every year in the US from self-driving cars, too?

For that matter, I don't think the cameras installed in Teslas work as well as a good human eye, with dynamic range and depth perception.

So it seems like a bad premise to start with, and the proof of the pudding is in the eating, so to speak. So many years in, and when they pulled the radar out for supply chain reasons they had to disable self-parking, too.

The majority of road accidents are caused by distractions and inhibition. If you perform like the best drivers on the road, then you won't have accidents.

That last statement is an assertion lacking evidence, and as counter-evidence I offer every accident Tesla’s “autopilot” has been involved in.

If drivers always paid good attention, I’m sure we would have fewer collisions. I’m not sure we would have none.

In such a safety-critical system, it's important to have backups. You need at least two cameras to accurately estimate depth, and cameras can fail for a variety of reasons (sun glare, low lighting, heavy fog). Radar, at the very least, is a backup for the cameras. Also, with vision, the best you can do is estimate distance, whereas with radar and LIDAR you are explicitly measuring it.

>Also, with vision, the best you can do is estimate distance, whereas with radar and LIDAR you are explicitly measuring it.

But is there any evidence you need to measure distances? We humans can navigate the world without walking into walls, so long as we're looking where we're going. For a machine to navigate the world it should be possible to do it via vision. And Tesla's do have multiple cameras to be able to measure depth.

And radar is not a backup for cameras. The resolution of the data is terrible and you can not rely on it to do any sort of driving except braking if it thinks there's an obstacle. Radar is also susceptible to problems as well, which is why Tesla's and other cars with radar can often go crazy thinking you're gonna crash randomly.

>The argument from Andrej Karpathy is that humans use vision and vision alone, so if we can learn to drive by vision, then so should an ML model.

Yes ... and? Is this guy like an oracle with all the right answers wrt to AI?

You're also appealing to the authority of a guy that didn't deliver much in 4 years and then just left the company. Then after he leaves Tesla brings back the radars lol, but that's definitely a coincidence, right? Because he's Andrek Karpathy, and he was definitely not wrong about that, at all.

I'm not appealing to anyone. I'm just stating his argument since he was one of the main guys at Tesla working on this problem.

He might have a point, he might not. I stated the obvious problems with it in my original comment.

> The argument from Andrej Karpathy is that humans use vision and vision alone, so if we can learn to drive by vision, then so should an ML model.

By that argument, we should abandon silicon for our chips, because we think without it. Heck, by that argument, we should be building androids and putting them into the driver's seat, because humans use hands to drive.

What an incredibly stupid argument. My respect for this guy just went right into the toilet.

>By that argument, we should abandon silicon for our chips, because we think without it

Evolution has created some of the greatest computational machines. If it was possible to build a programmable biological brain, we'd do it in a heartbeat if that's what it took to develop true artificial general intelligence.

>Heck, by that argument, we should be building androids and putting them into the driver's seat, because humans use hands to drive

Humans don't need to use hands to drive though. We could input steering and throttle controls though many different ways. We have vehicles that are controlled by tilting our bodies in different directions. But the fact of that matter is, that we use vision to see where we're going and the infrastructure is designed for vision.

I'm not sure what your point is, it doesn't seem like you understand his.

> The argument from Andrej Karpathy is that humans use vision and vision alone, so if we can learn to drive by vision, then so should an ML model.

Humans also learn social dynamics through numerous other senses and in situations that Tesla's models never encounter or code for. Those other cars on the road do have human drivers, after all.

Humans also have ears, etc.

Plenty of unusual road or lot situations where those things come in handy.

Everybody got bullish on this when Google first publicly presented their autonomous car and everybody from IT said: 10 years from now.

Except all the automotive people said: Nay, way to complex to solve.

The automotive guys worked since the 1960s/1970s on this. Yes it got boosted by the advances in AI, which where mainly driven by the now available computing power/memory. But AI and its limitation is known since the 1980s/1990s. I learned AI in hte mid 1990s in the university as part of my degree as an electrical engineer.

Now, we are at a point where even the Waymo guys (former Google Car) admit, that the fully self driving car will be years away. It will take much more years then everybody expected. The cars the Waymo runs, are running in very dedicated areas, which are highly mapped. They still have problems with temporay obstacles, like road construction. They still have problems in unexpected weather situations.

So it is, that even all the tech people who believed in the sayings from Elon, now realizes that its just vaporware.

It's not vaporware, it's just taken significantly longer than expected.

Tesla's solution looks more like vaporware because of how many times Musk has promised fully driverless operation -- sometimes even promising a cross country hands off trip -- within a year or two. Other companies have also had overly optimistic predictions, but generally not "we'll be able to go anywhere in the country with no intervention within the next year" level of extreme.

Waymo's progress has been slow relative to "ten years" predictions, but they've also made steady progress, and it looks like it's continuing.

CPU/GPU processing power is clearly making steady progress and looks like it will continue. Yet nobody thinks we will be able to brute-force 2048-bit RSA with the best sieve techniques even if the progress continues for 100 years.

It is clearly not sufficient just to have some measure of growth to achieve some goal. It is necessary to have the growth actually matter compared to a measure of the problem difficulty. I would argue for self-driving cars as in Level 4/5 autonomy, nobody even has any useful measure or understanding of the problem difficulty.

It's really easy to understand the autonomous driving levels. It's also really easy to understand the Kardashev scale. It might just turn out that both are of equally little practical relevance in our lifetimes.

I know this is a boring situation, but there is no guarantee the universe shouldn't be boring.

Waymo has an operational SDC taxi service in Phoenix with no safety drivers. Obviously that's a fairly easy metro as far as driving goes, but it's enough progress to be long term optimistic imo, especially considering they're doing a closed beta in SF currently.

Tesla is also selling "full self driving" as a feature, right now. Not "coming in a year or two" They are taking people's money for it right now, and have been for years.

It's not vaporware, it's just taken significantly longer than expected.

EDIT: nevermind, parent clarified. Leaving original comment so follow-ups make sense.

"Taking longer...": what we call it when it doesn't work, despite all of the marketing. So, IOW, "vaporware".

When Tesla ships a working product, where "working" something remotely close to what has been promised, we can call it something else.

Sorry, I interpreted GP as saying that self driving in general is vaporware. If they meant specifically just Tesla is vaporware, then yeah I largely agree.

Oh, okay, I get ya. Yeah, I could see self-driving being a thing...some day, and wouldn't put the whole thing in the vaporware bucket. Thanks for the clarification.

Robotaxis are on the road today in multiple cities. That's nothing to sneeze at, large fleets of totally autonomous vehicles in complex city environments is something something roboticists have never done before!

Those environments have been mapped out to a very high level of detail. It's a similar "long-tail" problem — all very well and good to have self-driving cars running around downtown SF after spending tons of effort and time creating a virtual map of the city, but it's quite something else to get general self-driving that will work across the entire country.

Cruise went from nothing to a public driverless service in Austin in 90 days, and that involves a LOT more than just mapping the area. Waymo had a comparable timeline for expanding to downtown Phoenix from the suburbs. Mapping is not the albatross you think it is. It's essentially just Google Maps/Streetview (which covers the whole world) with extra sensors and larger hard drives. A problem that's been solved for over 15 years already.

I don’t think doing it for the entire country is even a goal for these AV only companies. For trucking theyll HD map the highways. For city center taxi service HD maps and remote assistance are fine.

Did you read and understood what I wrote: "The cars the Waymo runs, are running in very dedicated areas, which are highly mapped."

I simply don't see the problem with areas being highly mapped. For the record, operational areas can be mapped very quickly now - I think I read a Waymo presser that said they mapped their LA operational area from 0 in a few weeks.

    Musk is distracted and showing that he 
    doesn’t have all the technical chops we 
    thought he did (if reports from inside 
    twitter are to be believed)

Who thought he had technical chops? I don't mean this as snarky or glib. But it was surprising to read.

The rosiest take is that he's a Steve Jobs type as opposed to a Bill Gates type. A guy who assembles and rallies teams of engineers and designers around a singular vision and often manages to pull, push, or drag them across the finish line.

I'm not a fan of Musk in particular, although I am also not demeaning this sort of CEO. The ability to form/lead/motivate teams that execute on this level doesn't exactly grow on trees. And it's all but impossible for a CEO to have deep, current "technical chops."

It's just that I've never actually heard Musk credited with having technical chops.

> Who thought he had technical chops?

Many, many people liked Musk, or called him more credible, for being the one CEO who also knows engineering. It also made him aspirational or a role model for many engineering types. It's a bit like when people mention that Angela Merkel has a PhD in quantum chemistry, it confers legitimacy in context.

The difference, of course, is that Merkel also actually has her degree.

Musk regularly repeats that he is an "engineer, not a businessman"

This video edits together examples from various interviews: https://youtu.be/e7ez_WF40hY

Ah, okay. That explains some things. He's the one saying it, and I guess some folks bought into it and believed him.

> But the long tail is killing them.

I imagine their engineers have said they’re “90% finished” once or twice in the last few years…

Always a red flag!

To be fair the last 10% of any project is generally 90% of the work. So saying you are 90% done yet being far off is truthful, in a sense.

As a leader of software teams at various points of my career, I have to completely reject this notion.

In a poorly managed project, yes, the last 10% of a project is 90% of the work. A healthy project puts 90% of the work at the front: the work of planning, narrowing scope, thinking creatively about what could go wrong, leaving oneself off-ramps in case of unexpected situations. The speed or cadence of a project should only increase as you near release; momentum should build. There should be a natural period of "dust settling" before production, rather than scrambling to pull the system into a healthy state.

If an engineer repeatedly says they're "close," but cannot quantify such a statement, I find they're (generally speaking) completely full of it.

Perhaps Tesla's situation is unique (or perhaps not). Musk has repeatedly said it's difficult to gauge completion date because no one has built this software before, so there is no benchmark. Having just sold my Tesla in favor of a competitor, I suppose I won't be there for the release anyway!

I think it depends on the project. 90% of SWEs are using pre-packaged databases, analytics, logging, APIs and whatnot. There is no 10% or 1% tail bogging them down because they're just configuring tools. But if you're in the business of innovation, it's really hard to know what lies in the darkness ahead, and it it very common to attack the easiest parts first, and what you thought was a 1% that would take up half the time is a 1% that cannot be completed ever.

I think the point is that if that final 1% is essential to the project then the project was poorly planned (like the comment you replied to said). Why spend untold time tinkering on the fun easy stuff ignoring a 1% that could kill the whole thing? If that 1% isn’t essential then you should just not do it — and if you don’t make that decision then the project is also poorly planned.

I think if every single great discovery in the course of humanity had to have been "well planned," they would have never happened. Like I said, most of the "disruption" and "innovation" happening in the startup ecosystem is far from. And projects of that scale involve little to no unknowns so long as a properly experienced team is attacking them.

But when you're talking about AV or fusion power (or even much smaller discoveries that sit on the vanguard of human knowledge), it requires the kind of leap into the unknown that all great discoveries have necessitated.

> To be fair the last 10% of any project is generally 90% of the work.

No, that means a pathologically badly planned project. If some part of your estimates are off by an order of magnitude, you need to learn to do a lot better at estimation.

Is this a Zeno's paradox, where you can never actually finish?

Have heard a twist/joke on this where an engineer says something like "i'll never get all the way there but close enough for it to work"

Its dishonest to not scale by effort required when reporting project progress.

The thing is human intelligence shines in the long tail and stochastic parrots thrash and die there.

This is absolutely correct and I'm not sure why people are downvoting you, unless they're not familiar with the stochastic parrots paper. https://dl.acm.org/doi/10.1145/3442188.3445922

> I'm not sure why people are downvoting you

I am not sure too. I also want AGI to happen but we need to acknowledge the limitations of current formalisms and I think people are married to formalisms.

Isn´t what they have right now safer than human driving?

They haven't demonstrated that in a like-for-like comparison. Tesla's autopilot, in circumstances where it can be engaged, and with human supervision, has more miles between accidents than average human drivers in all circumstances (such as older cars, worse roads, and worse weather). But in the circumstances where Autopilot can be used humans are also much better than that average.

From watching a number of videos, I'm also under the impression that Autopilot is currently still so unreliable that the average beta tester of the system is overall _more_ attentive (and often more stressed) to the road than they would otherwise be, so they can intervene in questionable cases or even just override otherwise embarrassing road maneuvers.

Given that the typical Autopilot ride has multiple disengagements that also makes it really hard to judge the performance of the system properly - if you always have to have a hyper-ready human waiting in the wings to catch mistakes to produce nice stats, that's certainly still far away from what most people want in such a system. You'd need to demonstrate better safety without disengagements and without a human backup, as the human drivers you compare to don't have one, either.

Marques Brownlee demonstrated this nicely in a recent video review of his daily work commute using Autopilot.

I believe you're confusing autopilot with FSD beta.

I did, even though the video I cited taught me the difference. Thanks. :-)

Safer than human driving isn't the bar that needs to be cleared. I consider myself a much safer driver than the average human (which also includes 70 year olds with driver's licenses) and I don't drive when I'm sleepy or drunk or even a little impaired. Even if the claim that it's "safer than human driving" is true, I would only ever use it if it was safer than my driving.

Fun fact: most people believe they're a safer than the average driver. Many people even believe this while driving drunk. Unless we start putting interlocks on all cars (which isn't entirely hypothetical[0]), they're just going to keep driving around drunk. Would you rather they drive drunk, or have an imperfect system that's better than drunk them, driving a car that also has an AEB (Automatic Emergency Braking) system?

[0] https://www.motortrend.com/news/anti-drunk-driving-technolog...

I know I am not as good as the average driver, simply because I have much less practice, but I make up for it by extreme vigilance and caution, and I've never been in any sort of accident or moving violation.

I think the world would be a better place if most people believed they were bad drivers like I do, and were much more vigilant and less aggressive.

Practically speaking, even if self-driving technology was widely available today, it would take decades for manual cars to get off the road.

And of course the pragmatic reality is that nearly everybody considers themselves "better than average."

Tesla isn't merely competing with the actual average human driver, they're competing with the average human driver's unrealistically rosy perception of their own driving prowess.

It's not necessarily unrealistic. It only takes one terrible driver to drag down the mean and make everyone else better than average!

A good example of how nearly everyone can be above average is number of arms. If, like 99+% of the population, you have two arms then you have an above average number of arms.

Yeah, no.

Driving skill is almost certainly normally distributed, and if so, your answer is wrong.

From personal experience, I can say what they have now drives worse than a drunk teenager.

This is the type of question that will start arguments on both sides, but the truth is no one outside Tesla could possibly have the data to say one way or the other.

What they have now is a small subset of "driving" so not really useful to compare as it's misleading at best. Otherwise you could say student drivers are the best drivers since they almost never cause accidents.

Hard to say. Are there published comparisons to affluent humans driving relatively new cars manually?

Downvoted for doing a question that nobody denies. I love Hacker News these days.

In a nutshell, the article says adjusting for sales volume, Tesla auto pilot crashes numbers are far higher than the ones from Ford.

"...Tesla ought to have a NHTSA-reported crash total of 70 since last summer to be comparable with Ford’s rate. Instead, Tesla reported 516 crashes."

I wonder if Tesla drivers simply use autopilot more than ford’s do.

Considering Tesla's deceitful marketing, that is likely, yes. But Tesla/Musk shares the blame for those accidents because of the false advertising.

Probably they do because of false advertising

Curious to know where you see this false advertising?

When you go to purchase Autopilot on the Tesla website [0], it says in the first sentence: "The currently enabled features require active driver supervision and do not make the vehicle autonomous."

On Ford's website regarding their Blue Cruise[1]: "Blue Cruise allows you to operate your vehicle hands-free while being monitored by a driver-facing camera to make sure you're keeping your eyes on the road."

If anything, Ford seems to be more bold and brazen with their approach here.

[0] https://www.tesla.com/model3/design#overview

[1] https://www.ford.com/technology/driver-assist-technology/#bl...

The fact that on Tesla's site the feature is called "Full Self-Driving Capability" while Ford's is called "BlueCruise" seems to be a rather relevant difference. "Full Self-Driving" sure feels like false advertising to me, even if it doesn't legally qualify as such.

Mixed signals

https://www.tesla.com/autopilot

"Tesla cars come standard with advanced hardware capable of providing Autopilot features, and full self-driving capabilities—through software updates designed to improve functionality over time."

"Current Autopilot features require active driver supervision and do not make the vehicle autonomous."

"The system is designed to be able to conduct short and long distance trips with no action required by the person in the driver’s seat."

Here’s the part I see as false advertising:

https://www.tesla.com/autopilot

> Tesla cars come standard with advanced hardware capable of providing Autopilot features, and full self-driving capabilities—through software updates designed to improve functionality over time.

There is no way for Tesla to know that their cars have the hardware needed for full self-driving capabilities because they don’t know what hardware is needed for that.

Also the branding of their “Autopilot” with a capital A allows them to make up whatever definition they want for it and lets customers (understandably) confuse it with the standard definition of “autopilot” without a capital A.

They name it "Full self driving" and lean heavily on that marketing for one: https://www.tesla.com/en_eu/support/autopilot

> Curious to know where you see this false advertising?

Musk has been lying about full self-driving for years: https://jalopnik.com/elon-musk-promises-full-self-driving-ne...

Tesla has already been done for false advertising once in court. Tesla didn't even try to argue the case: https://electrek.co/2022/12/12/tesla-ordered-upgrade-self-dr...

Now Tesla is hoping to avoid being done for fraud by calling their "full self-driving" a "failure" instead of an outright fraud: https://edition.cnn.com/2022/12/12/business/tesla-fsd-autopi...

But with years worth of lies behind it, and especially because they've been selling it to customers and still haven't delivered it, it's tough to make the case that it isn't fraud.

I'd wager yes, based on the promises Tesla has made about the feature compared to how Ford talks about theirs.

Legitimate question.

Ford’s Bluecruise is far more limited than autopilot. It’s geo-fenced to some freeways.[1] Autopilot works on any road with lane markers. My guess is that few Ford drivers use the feature.

1. See the map on this page: https://www.ford.com/technology/bluecruise/

What a strange comment,

It seems like you're trying to make a point for Tesla but the arguments you present make Blue Cruise seem like a more carefully planned and safer technology, lol.

I'm saying that almost nobody uses Bluecruise because it's a gimmick. It only works in a few places. In the places where it is supported, it refuses to turn the wheel too sharply, forcing you to turn in areas where there is a significant bend in the road. If you make your feature so useless that nobody uses it, of course that feature won't cause many deaths.

The useful metric is deaths per distance traveled regardless of whether automation is on or not. Otherwise you can Simpson's Paradox yourself into any conclusion.

I promise you they use the feature. It’s quite nice. People want it enhanced, but it is useful.

But as a sibling comment pointed out it’s limited for a reason. It’s carefully locked to areas Ford has mapped and verified and believes the software/hardware is capable of performing safely. It will disengage for anything more complicated.

Tesla’s “you can run it almost anywhere, good luck” attitude could easily help explain a large chunk of the discrepancy. The Ford won’t engage where it’s unlikely to work while AP very well might.

The comparison is based on number of enabled vehicles and not on a mile-for-mile basis.

I'm kind of coming around to the idea that Comma AI's approach of full end-to-end is the right one.

Tesla, Waymo, Cruise, etc all take real world data and load it into a virtual representation, then do planning inside virtual space, then execute that plan in the real world. It seems like this is just really clunky and doesn't handle weird edges very well.

Comma AI's thesis is that creating this big virtual world to do planning in is both a waste of time and compute, and also a worse driving experience. They just take a ton of human data, and essentially say "given this situation, what would a human do?". I.e. it's a neural network that takes in the last few frames of video, and outputs steering + accelerator + brake commands, with no intermediate virtual space representation.

A huge advantage of this approach is it scales really well. Human data in, model out. They are maybe a year behind Tesla FSD in terms of capability with a tiny team of ~15 or so, and they are gaining ground.

Tesla could probably improve their system by adopting a similar end-to-end approach, but I think this may not be easy to do for organizational/corporate political reasons. They have several hundred engineers working on all of the various bits of their FSD stack who are all incentivized to defend their turf.

This is not good enough to handle edge cases.

It's a little bit like ChatGPT: ChatGPT is a next-word prediction engine. It has ingested and analyzed "human data" which has given it a take on what might come after a prompt. With good training data, a large enough token window and a helping of RLHF this can do very cool things. But there's no actual reasoning going on.

An "edge case" is a prompt that throws this approach for a spin, because statistics over the training data corpus don't yield a good prediction on it. For example because it's novel or its structure is uncommon. In those cases ChatGPT will do its best and confidently return something very dumb.

With the driving problem, Tesla and others are trying to mitigate this with two approaches:

- superstructures of rule-based planning to keep the predictions within safe lanes (no pun intended), i.e. bake in some reasoning after all

- trying to write simulations that auto-generate "edge cases" or just stuff that is underrepresented in the training data set and add those simulated 3d renders to editorialize the predictions

It's currently still unclear if this is enough to solve the problem or if we're still some innovations (e.g. ones that get us closer to AGI) away from being able to make it work.

I wonder if self-driving doesn't actually require reasoning, though. Most of us are able to drive instinctively, without fully engaging all of our attention; we certainly aren't solving problems with deep analytical thinking. Perhaps it's a problem that's well-suited to the approach an AI like ChatGPT takes?

> Most of us are able to drive instinctively, without fully engaging all of our attention; we certainly aren't solving problems with deep analytical thinking

Depends. Many of the type of situations where FSD runs into trouble or disengages are also the ones where a human would slow down/pause and ponder a moment what to do next. In the FSD case, it then often eventually gives up and disengages, because it can't confidently predict what action to take. Humans show better performance in those cases.

If you ask me what separates a human from ChatGPT right now, it's that ability to use reasoning to fill in for bad training data. If, as some people have posited, the training data for ChatGPT-like systems is possibly already exhausted, we need some new ideas.

But humans are good at that. :-)

Aside from that, note also that your learned instinctive predictions take into account many more parameters than the models a Tesla currently runs anyway. For example you have an understanding of social dynamics (so what those other human drivers think and what might be acting on them, etc.) that is far in excess of what the Tesla models can learn implicitly from videos that observe them driving. With some of these things we have no idea yet how to feed them into the models.

I agree. I go weeks at a time without having to do any critical thinking whatsoever.

There probably are some situations that do actually require a looping reasoning process that a simple feed-forward neural network can't do, but they are fairly rare.

The most common ones probably actually involve complex parking lot navigation tasks, where you have to negotiate the space with other drivers, understand their intentions, yield space, take space, etc.

You may end up using reasoning more often than you think. I run into situations that require my full attention and are very likely un-AI-able once a week or so. (And I live in California.)

>This is not good enough to handle edge cases.

Is Tesla doing any better, though?

At least comma can handle the "huge unmovable object in front of me, I should brake" edge case.

OP claimed that Comma's approach would be able to handle edge cases better, but I don't think it's fundamentally a solution to that problem. Tesla is struggling with the same tech limitations certainly.

I think it will handle edges better because it's explicitly not trying to classify everything into objects (for the purpose of loading into their virtual representation).

So it doesn't need to "understand" a situation in terms of what is physically true, it just needs to understand what information is valuable for the task of driving.

Consider this edge case: https://www.youtube.com/shorts/kM-xBgz26pE

A utility truck carrying a load of new traffic lights. Tesla correctly identifies them as traffic lights and loads virtual representations into its world model. It doesn't seem to cause any weird behavior (thankfully), but this demonstrates a case where doing this fancy classification actually makes your system less robust.

I'd be interested in a direct comparison of how Comma's models handle the same situation. Has someone tried?

I know, and I'm saying that whatever Tesla is doing, along with those "mitigations" you mentioned, is crap because of the several crashes where it has failed to detect big stationary objects in front of the car.

I haven't seen comma doing stuff as dumb as that, so I'm also betting on comma's approach.

I have no stake in the Comma vs. Tesla, I just don't think the Comma approach can get us all the way to a full solution either.

What I think is interesting is that if you assume that there's still a couple of new ideas left before the whole thing can work, it matters who is the most poised to adopt (or even to have) the new ideas and turn them around into a product.

I also don't know who that will be. A vertically integrated company like Tesla? A tech supplier like Waymo? A nimble startup like Comma?

As a Comma owner (2 Commas in fact), I feel like Comma's main innovation is only releasing features (on the main branch at least) that they feel are quite battle tested, and at the same time not being afraid to lean into the fact that Comma is somewhat limited and isn't full FSD.

My Commas are bulletproof at doing what I expect them to do, and they don't do what I don't expect them to do. And from my perspective as a driver, it's incredibly, incredibly useful. When driving is easy and mundane I can trust that Comma can handle it (and Comma will tell me if it can't), and when driving is hard then I don't expect Comma to do handle every complex situation (though it does do a good job most of the time). They don't overpromise, and they deliver what they promise.

I'm not convinced this method will get to FSB, but I from my perspective I don't think that is necessary. I genuinely believe if every car on the road only had Comma's capabilities and no more, that would on its own reduce highway accidents by like 50%. I really wish some car manufacturers would lean into this and work with Comma on making a first-hand integration. I would 100% buy any type of car from any manufacturer if it had first-hand Comma support.

>>takes in the last few frames of video

isn't that how uber killed a woman walking a bike in phoenix? During the critical last second(s) the model didn't know what she was and kind of ignored her even though previous seconds had tagged her as a moving cyclist.

From the wiki article about the Uber incident:

> The recorded telemetry showed the system had detected Herzberg six seconds before the crash, and classified her first as an unknown object, then as a vehicle, and finally as a bicycle, each of which had a different predicted path according to the autonomy logic. 1.3 seconds prior to the impact, the system determined that emergency braking was required, which is normally performed by the vehicle operator. However, the system was not designed to alert the operator, and did not make an emergency stop on its own accord, as "emergency braking maneuvers are not enabled while the vehicle is under computer control, to reduce the potential for erratic vehicle behavior", according to NTSB

So Uber had a lot of systemtic issues leading to the crash, but even if most of those were fixed, I think Comma's approach works better here because it is explicitly not trying to do any of this fancy classification and prediction. If a human sees an unknown moving object near the road at night, they start slowing down even before they have identified the object. Comma is trained to just do what a human would do, so it will also slow down.

[1] https://en.wikipedia.org/wiki/Death_of_Elaine_Herzberg

Jesus that's right, I forgot they had turned off braking, fair enough to the Comma folks.

> takes in the last few frames of video, and outputs steering + accelerator + brake commands

Isn't this pretty close to what Tesla does?

Cars phone home with timestamped video from multiple cameras plus data on what hundreds of thousands of human drivers do, also timestamped.

Use all the human events to train a DNN to make similar decisions as humans.

Then augment that with GPS, maps and routing to a destination.

No, Tesla does something much, much more complex. They build an entire 3D representation of the world and do path planning inside that virtual model. Check out the videos from the last Tesla AI day. They are creating a voxel representation of the world, doing object classification and loading objects into the world, doing monte carlo tree search path planning in the virtual world, etc.

I'm not sure where you heard this, but you heard wrong. Tesla absolutely does use real world data to train their neural nets.

I think you have misinterpreted my comment.

Of course Tesla uses real world data. My claim is that their self driving system it not end-to-end. That is, it's not "just" a neural network that takes sensor data as input and gives steering/accelerator/brake commands as output.

They take in data from all their sensors, fuse it together, and create a real-time virtual copy of the world. They do planning inside this virtual copy of the real world. This has been demonstrated in many Tesla presentations. The FSD visualization on the screen of the Tesla is itself a depiction of the virtual world being constructed from the sensor data.

"Ogan, using NHTSA crash numbers, Tesla’s previous reports, sales numbers and other records, concluded that the number of reported Tesla crashes on U.S. roads has grown far faster than Tesla’s sales growth. The average monthly growth in new Teslas since NHTSA issued its standing order was 6%, he figures, while comparable crash stats rose 21%."

"The Tesla Autopilot crash numbers are far higher than those of similar driver-assistance systems from General Motors and Ford. Tesla has reported 516 crashes from July 2021 through November 2022, while Ford reported seven and GM two. To be sure, Tesla has far more vehicles equipped with driver-assist systems than the competition — an estimated 1 million, Ogan said, about 10 times as many as Ford. All else equal, that would imply Tesla ought to have a NHTSA-reported crash total of 70 since last summer to be comparable with Ford’s rate. Instead, Tesla reported 516 crashes."

> Tesla crashes on U.S. roads has grown far faster than Tesla’s sales growth

A fairer comparison would have been [ total number of reported crashes ] / [ total number of cars on road ].

Comparing it with sales growth will show poor performance for older brands which have a lot of cars on the road vs newer brands that are just release cars now.

Really needs to be number of crashes / miles driven.

I suspect the real difference is that Tesla drivers cede far more autonomy to the car than other drivers. When you look up from your book and see that you are 20 feet from a collision at 65mph, it's too late to do anything about it.

I wonder why they would do such a thing?

Perhaps dropping $10k or more on something called “full self driving” makes said drivers believe that the car can fully self drive by itself?

> I wonder why they would do such a thing?

For me it would be having used "full self driving" mode, diligently, for X years or X miles without any issues.

...he said that “solving” FSD is “really the difference between Tesla being worth a lot of money and being worth basically zero."

Why? As a consumer, I would be thrilled with an affordable vehicle with a powerful and reliable electric drivetrain, top-of-class range, comfortable features and sensible UI, and flawless fit-and-finish.

That should be what the company strives for in the medium term. Stop the fixation on tech that's a decade+ out and magnet for lawsuits and penalties from regulators. Make the best made-for-human vehicles out there, and become the most successful auto manufacturer. Keep working on moonshots but treat them as such.

Yes. Nonsense line. EVs and self-driving are orthogonal concepts. The only reason to combine them is to make a cute story for gullible buyers of Tesla cars and Tesla stock. If you assume that he knows that we know that, then he is simply pumping confidence in their ability to do self-driving. "We are all definitely drinking the cool-aid here, folks!"

Because once you have a platform for making affordable EVs, making EVs alone is a race to the bottom. Same thing as smartphones. They need moats in other highly-valuable areas to keep that share price high.

> I would be thrilled with an affordable vehicle with a powerful and reliable electric drivetrain, top-of-class range, comfortable features and sensible UI, and flawless fit-and-finish.

But Tesla is not affordable, and doesn't have flawless fit-and-finish

This isn't really here or there, but I've recently been going through the deeplearning.ai course by Andrew Ng and friends, and at the end of each week, there is an interview with a luminary in deep learning.

A couple of weeks ago it was Andrej Karpathy. I got about three sentences in when I realized this guy is really, really smart. The way he spoke about neural nets and the problems he was working on suggested to me a deep and nuanced understanding, and a way of thinking that always tries to expand that depth and breadth.

Anyway, I figure if a guy like that couldn't make it work after so many years, even with a team that surely has other strong players, then it's just out of reach for the time being, with the hardware they're constrained to. It's even possible that deep neural nets will just never be able to do FSD at a level that will gain broad acceptance and some new architecture will be necessary.

> I got about three sentences in when I realized this guy is really, really smart. The way he spoke about …

As a non-expert, how would you be able to judge?

ChatGPT also sounds really smart while giving brutally false answers.

Well, if you'd permit not getting into details, I'm not exactly a total non-expert.

Ah, why so gloomy. Solving this probably comes down to 1) sensors, and 2) computational power available in a car. The sensors used in Teslas were a joke last time I looked (low res, bad low light performance, not enough cameras probably), certainly we can do better? And Moore‘s law is still alive in a way, allowing stunning progress like demonstrated in Chat-GPT. Ever-growing car batteries will also allow for way more power draw for computing. I expect vast improvements in the next few years. Maybe we can get from „drives like a drunk teenager“ to „drives like an overly cautious grandparent“ at least.

> 1) sensors, and 2) computational power available in a car.

Waymo and Cruise use dedicated hardware and aren't artificially constrained either by preexisting sensors nor compute. Yet they haven't fully solved the problem yet, with Waymo still struggling with unexpected but really should be expected stuff such as road construction, while being available only in specific geographic areas and with quite expensive sensors and years of work.

Care to link the interview? I love Karpathy's work.

https://www.youtube.com/watch?v=xxu4IqwKw0w

Dang, I did not realize it is from 2015.

I was a HUGE shill for Tesla's vision (no pun intended) of full self-driving. However, between the (IMO) extremely short-sighted decision of removing ultrasonics to cut costs ahead of Vision replacing them, Elon's ridiculous antics over at Twitter, unjustifiable price increases of both their hard product and FSD, and, most importantly, FSD Beta being downright dangerous to operate, my faith in Tesla solving autonomy has all but evaporated.

It wouldn't surprise me at all to hear that Tesla is doing funny accounting with their Autopilot crash data. I'm starting to think that the feds will drop the hammer hard on their autonomy pursuits in light of this and will force them to:

- Change the product name to something less misleading (While I'm okay with ambitious product names like this, SO SO SO MANY people are not), and

- Honor customers wanting a refund (and I bet there is a long line of people that want one)

I've sold all of my shares in Tesla (admittedly, I didn't have many) and have not only cancelled my interest in another Tesla but have considered trading in our Model 3 for another EV despite a smaller charging network multiple times this year. (To me, there's no point in owning a Tesla if their supercharging network will be forced open to everyone and their path to autonomy is infeasible.)

I've been using FSD Beta in our Model 3 this past year. It has gotten a lot better throughout that time frame, but it is very far from safe to use, and I certainly wouldn't recommend it for anyone but the faint of heart.

When it works, it works really nicely. Beta has smartly routed around construction and does a good enough job of navigating around other cars and predicting their behaviors.

Unfortunately, when it doesn't work, it REALLY doesn't work.

Even in the most recent release that dropped this month, Beta has attempted to run red lights and drive on wrong sides of the road, has missed or made wrong turns a surprising number of times, and done some of what I can only call "generally extremely wild shit" (like super sudden stopping and sharp turns without telling you why). It usually picks lanes correctly, but will just as often pick lanes that will take you to the wrong place or straight up don't make any sense. It also LOVES switching lanes at the very very last minute, usually right before a turn onto another road, which is confusing for everyone.

It's actually gotten worse in situations previous versions excelled in.

I even turned it off completely at one point because it was about to miss a turn that it nailed several times before, in broad daylight, with no other traffic around, for no reason.

Recently, I've been thinking about GM, Ford and Waymo's approach to autonomy (LiDAR for object measurement and detection and pre-mapping as many roads as possible). If Google Maps could map over 80% of America's roads within two or three years, then assuming that mapping can be done "at the edge" (i.e. with regional mapping vehicles), what's stopping these companies from doing the same thing? Furthermore, if mapping and equipping cars with LiDAR scales well, then what worth is there in trying for a vision-only approach?

Welcome to where many of us have been for years! I'm glad you're driving more safely now.

It looks like you're more of a glass-half-empty person (and me a glass-half-full). I agree with your current assessment of how good or bad fsd beta 10.69.x.x is. However, as I see it, the rate of improvement is such that FSD will reach L3 autonomy in 2 years (and requiring one hardware update).

Especially because perception, which I guessed would be the hardest part to solve due to lack of LIDAR, seems to be pretty good already, it's the planning that's bad.

I was absolutely a glass-half-full person when it came to FSD (the insane amount of downvotes I've gotten on Reddit on FSD-related posts can back me up on that) until three things happened:

- Elon and his team committed the purge over at Twitter (which made me re-think his leadership abilities and doubt the credibility of his vision-only approach),

- Andrej leaving (why would the head of Autonomy leave when they are supposedly on their way towards completely changing how we drive? Makes no sense to leave your life's work like that), and

- A non-engineer, non-Tesla owning friend of mine got a ride in a Waymo in San Francisco and described it as a smooth ride start to finish. (I cannot imagine FSD Beta being smooth in a city setting; it sure as hell isn't in Houston proper; and Waymo is completely driverless!)

Now I'm definitely going to try it again when FSD 11 drops, but I am a little scared of single-stack given how absolutely rock solid AP is.

There are several FSD beta videos on Youtube (e.g. https://www.youtube.com/watch?v=FbH9GVcaEYM&ab_channel=Dirty...) that suggest no intervention. I'm curious why you dont think FSD Beta can be smooth in a city setting.

Re. Andrej - looks like he might be coming back and his reason for leaving was based on his role being too managerial shrug https://electrek.co/2022/10/31/andrej-karpathy-coming-back-t....

I wish that on these discussion, we could also see a report from commentors about how much time they have spent behind the wheel of a Tesla on autopilot.

It is truly, truly depressing reading these threads.

because it doesn't work, nor ever will...

LA Times reporter Russ Mitchell's entire career is mostly clickbait Tesla articles

He and Hiltzik really have it in for Elon.

Because all of their self driving stuff is falling apart since they dropped Lidar.

They dropped radar, not lidar. But apparently are bringing back a new and improved version with higher range and resolution. Last I heard it was supposed to ship in cars produced in Jan. They already filed the related FCC paperwork.

What happens to people who received cars without radar? Do they get retrofitted, or just keep using "full self driving" and hope for the best?

Dunno, mine had the Radar, I think it was disabled for awhile, and no idea what happens to my old radar when the new higher resolution radar becomes the standard.

It was Mobileye who dropped Tesla because Tesla was making claims Mobileye was uncomfortable with.

https://arstechnica.com/cars/2016/09/tesla-dropped-by-mobile...

Regarding autopilot from 6 years ago? Please. Fast forward, Telsa has built their own neural chip and vision system that surpasses Mobileye in production and scale. And their new HW3 chip can do 144 TOPS. Let's get real, Tesla out innovated Mobileye with their own closed loop vision system at production scale.

The 2016 mobileye summon feature worked. The current tesla smart summon feature is still in beta.

[citation needed]

They've never had lidar and no.. self driving isn't "falling apart". You have no idea what you're talking about.

You're right. Things have to exist before they can be considered to fall apart.

Never never had lidar.

Current autopilot is basically in maintenance mode for at least 2-3 years. They are probably preparing to switch to a single stack fsb beta 11 as it was announced. Could be the reason.

Reply on news.ycombinator.com