Musk Distorts Data: Tesla Still Years from Full Self-Driving

Individual driver experience is basically useless for assessing if it has "arrived" due to the fundamental lack of data any one human can generate, however individual driver experience is adequate for assessing if it has not "arrived".

To explain, suppose a manufacturer claims that their widget fails 1 in 1,000,000 times. Suppose a regular human can use 1,000 units. Even if a regular human finds zero failures in their 1,000 unit random sample that does not provide evidence that the manufacturer's claim is true. You need over 1,000,000 samples, usually on the order of 10,000,000 samples, and observe a number of failures comparable to the claimed rate to make any such claim in a statistically rigorous fashion.

In contrast, if 100,000 samples are collected and 10 failures are observed, a rate of 1 in 10,000 which is already a expected failure rate 10x more than the amount any individual person would use, then you can assert with high confidence that the manufacturer's claim is a lie even though you have not even observed 1,000,000 samples as would be required to prove the manufacturer's claim is true. If a regular human discovers more than 1 failure in their 1,000 units, a observed failure rate of over 1 in 1,000 then you can assert with extreme confidence that the manufacturer is lying. The bar for adequate data to establish statistical significance is the number of failures, not the number of units.

Individual drives are over a factor of 1,000x from establishing the requisite reliability level and thus have 0.1% of the adequate evidentiary power to establish success, but can be used to establish failure. Even a literal lifetime of human experience with zero faults is barely adequate to establish success, with even a handful of failures over a literal human lifetime being sufficient to reject the claim of a reliability level adequate for fully self driving operation outside of a testing protocol in the hands of consumers.

This is basic statistics and it is literally impossible that the people at Tesla do not know the level of evidence needed to support their claims is over a factor of 1,000x more than their auditable presented evidence. They just choose to deceive their customers to support their stock price and then blast soundbite arguments that sound good, but are intentionally deceptive, to overwhelm the discourse.

Personally, I don't think so.

There are simply too many driving situations where visual input is severely limited. FSD can use all the help it can get.

Do you think that Tesla has presented sufficient auditable evidence to support their conclusion? As the article notes, they have presented zero auditable first-party evidence of their claims despite copious data collection. However, as the article notes, the CEO has pointed to third-party data as representative of their claims of improvement. But, as the article notes, that data that very same CEO points to emphatically does not support the specific claims of the CEO or support their projections.

Are you arguing that the data/evidence cited is low-quality? Then the CEO's claims are also supported by low-quality evidence and thus the conclusion of the article, that Tesla has presented insufficient auditable evidence of their claims, is correct.

The only way the data in the article does not justify its conclusions is if the evidence is "high-quality" and supports Tesla's specific quantitative claims. I do not see how the data showing a 2.7x improvement is support for Tesla's claims of a ~5-10x improvement.

For that matter, the amount of data that Tesla would need to support their claim is so miniscule that it would be trivial for any half-competent testing process to to produce auditable statistically-sound first-party evidence of their claims prior to deployment and would be trivially verified during deployment within a week. They have chosen to present no such evidence for their claims instead opting to point at low quality crowdsourced third-party data in preference over their own despite the fact that such data disproves their own claims.

There is something to be said of pointing at third-party evidence supporting your claims as it minimizes the risk of of a conflict of interest and looks more unbiased. However, deliberately pointing at third-party data that disproves your claims? You would either not comment on it or release your own first-party data showing better results (which runs the risk of critics arguing it is biased). As I noted previously, it would be trivial to collect and present such first-party data both prior to and after release to demonstrate that their statements were not made with a disregard for the truth, and yet they have not done so.

So, we are left with three conclusions:

1. They do not collect first party data. Their claims are made with a disregard for the truth or falsity of the statement. That meets the standard for fraud and the second conclusion of the article.

2. They do collect first party data, but the data is worse than claimed and yet they state otherwise. That meets the standard for fraud and the second conclusion of the article.

3. They do collect first party data, and the data is equal or better than claimed. Despite that, they cite third party data that disproves their own claims because they are so humble. This coming from the CEO who routinely makes statements that are legally so outrageous and untrue that no reasonable person could believe them. Who has historically run smear campaigns [1] against whistleblower campaigns highlighting the dangers of Autopilot when Tesla thinks it can argue the other side. That company who publicly rails against "wrong" data is intentionally citing data that disproves their own claims while having better data that proves their own claims out of a sense of humility. Seems legit.

[1] https://www.tesla.com/blog/bigger-picture-autopilot-safety

I do. We knew better 10 years ago. Elon simply lies continually. That's probably why he and Trump get along so well.