Comma.ai, self-driving cars and indefinite optimism
mleverything.substack.com
mleverything.substack.com
Releasing “alpha” level software and saying essentially “lol have fun idk” when it can significantly impact customer safety is childish, bad business, morally wrong, and something we as an industry need to move away from. It doesn’t matter if your customer base are early adopters, hackers, or open source supporters. You still owe them their safety and a reasonable and coherent privacy policy.
I don't see the Ford complaint as them introducing features slowly and carefully. To me its a code smell. But my biggest complaint is that its opaque. If they had a detailed description of the features, what sensors are used and their limitations, then I would be much happier. But instead they have nothing more than a name and a trademark.
Again, the "alpha" level doesn't mean anything. Hotz is immature and some of his actions are regretful. It certainly doesn't live up to the thought and care he puts into his products. Of course, I don't know the guy but I have heard him speak for hours and was very impressed.
I can completely agree the naming and descriptions are horrendous though. I think that as more of these technologies start to provide assistance in more areas of driving, more of the time, education and transparency are nearly as important as safety. And that’s an area where every one of these projects is failing to a greater or lesser degree.
Consider BLIS (Blind Spot Information System) with Cross-Traffic Alert.
I imagine there's an engineer that created a function isBlindSpot(state, "right") as opposed to a more comprehensive system of object detection, where it'll be able to tell you if there's a blind spot but also be integral in Intersection Assist (a separate feature in the 2021 Mustang Mach-E).
Of course I could be wrong, and the features could be an interface to a more comprehensive system, but my hunch is that it's more segmented.
Automakers and Comma are operating at completely different scales. Orders of magnitude apart. You’re going to hear about more autopilot issues from major manufacturers because there are far, far more of them on the road. Comma only has a small group of loyal customers who make the leap to buy and modify their car.
Comma also incentivizes customers to keep quiet if the autopilot fails. The drivers can’t blame a multinational automaker if something goes wrong. They have to admit that they modified their car with an aftermarket device that hasn’t been safety tested. Comma even provides disclaimers that their product isn’t production ready. If it fails, the customer doesn’t have much incentive to broadcast to the world that they bought, installed, and used a product on public roads that was advertised as being not ready for prime time.
It’s false to equate Comma’s software strategy with that of major automakers. At minimum, automakers are working with safety regulators. Comma has gone to great lengths to specifically avoid dealing with anything safety related.
Super Cruise has 7 million miles driven [0]. Comma has 35 million miles driven [1]
> Comma also incentivizes customers to keep quiet if the autopilot fails
How do you figure? The amount of weird hatred people have towards comma is much more than Cadillac. You can find literally hours of real life video of comma driving in cars. Professional reviewers like Consumer Reports don't have a pro comma bias. Comma doesn't have a PR guy or teams of lawyers. They're not a hundred billion dollar company. They're a dozen or so engineers. They have no influence over trade publications or blogs. I think people and reviewers would judge comma much more harshly than they would big auto.
> At minimum, automakers are working with safety regulators
> openpilot is developed in good faith to be compliant with FMVSS requirements and to follow industry standards of safety for Level 2 Driver Assistance Systems. In particular, we observe ISO26262 guidelines, including those from pertinent documents released by NHTSA. In addition, we impose strict coding guidelines (like MISRA C : 2012) on parts of openpilot that are safety relevant. We also perform software-in-the-loop, hardware-in-the-loop and in-vehicle tests before each software release. [2]
Please point me to the safety documents of the competitors.Most don't even have driver monitoring like comma.
[0] https://www.cadillac.com/world-of-cadillac/innovation/super-...
[1] comma.ai
[2] https://github.com/commaai/openpilot/blob/master/SAFETY.md
>Super Cruise has 7 million miles driven [0]. Comma has 35 million miles driven [1]
Tesla has likely exceeded 4,000 million miles with Autopilot at this point versus Comma's 35. That certainly qualifies as "different scales" and "orders of magnitude apart".
It is also frankly bizarre that the author of this article barely even mentions Tesla while treating Cadillac and Audi as the primary competitors for Comma.
Comma is comfortably level 2 self driving, essentially lane keep. They don't detect red lights or stop signs, the don't have navigation or even maps. It just keeps you going in a straight line, with or without lane markings. It's much more comparable to something like Super Cruise.
My point was that lane keep assist technology exists as features in new cars. That's the target market and that's what comma should be benchmarked against. If you have a problem with comma's safety features you should have an even bigger issue with that of the large auto makers.
Well this is just terrifying.
I'm wondering if there have been any serious accidents where a driver was using his software and got taken to court over the repercussions of the accident.
Thinking about tens of thousands of people using this software on the road is pretty scary.
Personally, I can't wait for wider adoption of mature "assisted driving" products in the future - I find the idea of that future a lot less scary than the millions of not so great human drivers we have to deal with right now.
The biggest problem I have with Comma is that the quality of the camera installation will vary enormously between users even if we assume the software to be perfect. At least in a car certified for production you can mandate the cameras, sensors etc be attached permanently to the vehicle in relatively sound locations, vastly reduced risk of them falling off and causing a bad steering or other control input.
Plus it's a toyish feature
But there's no way I would bring this up in their GitHub, for example, because it isn't the culture of the project - and that's the problem. For example, their "leader" shows up and makes commits to their safety code like "WTF WHY WAS THIS SHIT EVERYWHERE."
To Comma's credit, they have made great strides in safety since the first two years of their product - since 2018, they've introduced safety checks that (while undocumented), seem to be somewhat consistent (or trying to be) across vehicles, are written in MISRA C which is at least something that's actually used in the field, and exist at the low level of the system. Commits are signed now, and their founder seems to make fewer non-documenting troll commits to the product. These are all signs that the company is growing up (and, if we look at the history of the company and their leadership, we certainly see some signs as to why this may be the case...).
But, these are things that should have been in place since day one, and the founders and company were certainly not unaware. It is absolutely fair to judge products and companies for the decisions they make, ESPECIALLY when their product is purportedly "world-changing," and in the case of Comma, safety does not seem ingrained into the decision-making process.
As an aside, this blog post is killing me, between extensive Peter Thiel quotes, random jabs at California's COVID policy, Elon worship, and a lionization of a startup company with an immature founder building safety-related hardware.
Every self-driving product that deploys to consumers is going to end up in court eventually. It's inevitable.
Imagine trying to defend comma.ai to a lay jury... the prosecution won't even have to try.
Meanwhile, taking these extra rote precautions (e.g. cut down on global state, throw unit tests on everything, don't use go-to's) has no real effect when the elephant in the room is that even correctly-implemented state-of-the-art machine learning models only have classification accuracies in the mid 90%s at best.
That's (relatively) fine if you're deciding what ad to show to a teenager using TikTok. When we are talking about driving a car, that loosely means having an accident every 100 miles. Based on publicly available disengagement data, this corroborates (only Waymo has surpassed the 1000 mile mark, GM might have recently, too).
Meanwhile, verification in this domain only means replaying actual recorded sensor data in a "simulation" using one-off example scenarios, not doing any kind of formal analysis or exhaustive proofs -- which simply don't exist yet for ML. There's no real way currently to take an arbitrary mis-classified example and determine its root cause -- or even correct it within the existing model without reducing accuracy elsewhere!
It's literally impossible to avoid being unsafe, so code style rules and procedures amount to mere hoop-jumping
I disagree, however that code style rules and procedures amount to hoop-jumping, as not every failure mode is the same (especially if you do accept the driver-attention / driver-in-control premise). An unexpected failure at the control layer (especially the control override layer) is a different class of failure to an unexpected direction command from the ML model, for example.
I don't think they have the money to fight multiple Court challenges, even if you're right you still need to fight. I doubt this company will be around in 5 years, and I at least hope anyone who installs this thing in their car and then gets into an accident loses their license.
I'm actually not anti-Comma, just responding to this specific point.
It seems that Comma are actively working on documenting their safety systems (via a new hire to own this task) and creating an audited checklist. And, there is a "new vehicle" checklist enumerated in a Medium post including some safety constraints, although in reading the Panda "safety" tree, I found no actual acknowledgement that these specific constraints were audited for each vehicle addition.
So, it seems that the great strides since 2018 continue - once these promises come true, they will have almost reached what I would consider the baseline level for a safety-related automotive system. Perhaps they will be able to secure an exit this way - I am actually rather surprised they haven't, given the froth level in this space as well as the valuable training data they have collected from their users.
Also, the author quotes Peter Thiel, claiming that the current cynicism towards self-driving technology is a type of "indefinite optimism" that hampers people's ability to plan the future. This is juxtaposed with the 1950's-1960's America, where Thiel claims "definite optimism" reigned. That type of indefinite optimism, at least in 1960's America, did not account for minorities, people of color, people of disabilities, etc. There was no future plan for them. Indefinite optimism, though it may appear lethargic and slow, at least is able to investigate the multitudes of edge-case scenarios that any $technology will face.
Arguably Comma is being a little more upfront about the user being a guinea pig for their software.
Exactly.
The Consumer Reports test wasn't a comprehensive safety analysis; it was a superficial "how will this feel to a consumer based upon a few drives" analysis. We are talking about a consumer reviews magazine... no regulator or jury is going to give a shit what a Microwave/Refrigerator review website said after performing 37 "tests".
In other words, the other products are "worse" because those are production-quality products for which automakers with substantial assets to forfeit are willing to accept a degree of real liability.
I'm sure if you compared comma.ai to GM's equivalent "alpha quality software" -- aka Cruise's L5 prototype -- the latter would come out way on top. But GM isn't stupid enough to ship alpha quality self-driving software, even as an aftermarket add-on.
Including comma.ai in this comparison was a huge oversight on the part of Consumer Reports. They should be comparing comma.ai to competitors' "alpha" products, which are all L5 prototypes.
"But most importantly let’s just stop being so derisive when it comes to world changing technologies." You just did it yourself.
I mean whose auto insurance(both collision and more importantly liability) has a clause that lets anyone hook up a random after market self driving toolkit to their car.
Who has liability once you start using their solution?
I'm sure comma.ai will have a clause that their solution is only for fun and you can't blame them if their self driving solution drives your car into a bus full of orphans.
So what's the point of what they are selling if you can't actually use it?
Or put another way, a self driving, or even driving assist add on only has value if the vendor absorbs all liability when using their system.
Ah, it isn't marketed as a full autopilot. I'm not 100% confident that it should be used in the current state but I'm also not an auto regulator, so we shall see.
Many car models limit the amount (and at what rate) that the computer system can turn the wheel. The system is piggybacking off of this engineering for most of its 'safety'. There was people spoofing park assist on some models to get very strong control of the wheel but such discussion got banned on the official discord.
Which leads me to my big problem with Comma.ai: it's open source but not very open. Look at the code, go back several releases and watch the number of comments regress.
There's zero documentation, the wiki is a fucking joke and for some reason does not come up in a google search and is just missing critical information. No, I don't want to watch a 5 fucking minute video to find a 10 second clip with the information I need. Write that shit down.
Instead you ask a question and some asshole that is gunning for a job at comma posts a passive agressive screenshot of them using the discord search. It's almost like if you put effort into having information recorded instead of being a dick you wouldn't have to be a dick. That wouldn't be very fun or inflate egos much so it doesn't happen.
Read through the Consumer Reports PDF [1]. It's abundantly clear that they have no idea how to properly evaluate adaptive cruise systems.
What this report tells you is exactly what you would expect from a Consumer Reports publication: the generic impression that a typical consumer will have after a few hours of use. This sort of superficial analysis is completely unrelated to anything approaching a real safety analysis.
[1] https://data.consumerreports.org/wp-content/uploads/2020/11/...
They even have a specific callout for Comma:
> A determination was made to include the Comma Two Open Pilot system manufactured by > Comma.ai. Although Consumer Reports does not endorse after-market modifications to all > consumers, we feel that it is important to include the test results in this report to the industry. The > direct comparison of this system to the other OEM systems will hopefully provide insight on this > alternative approach and highlight the areas across the industry that have room for improvement.
It is at least SOMETHING. I haven't seen anything else even close to as formalized that looks across all the various systems out there. Do you have better ones to look at?
This is a deflection. I'm responding to a comment that stated:
>> as they state on their website, they scored #1 for best at that by Consumer Reports compared to the stock offerings
CR is irresponsible for issuing ranked scores while saying in the fine print that these scores aren't actually recommendations.
That they didn't foresee comma.ai using their recommendation in marketing only makes them that much more irresponsible.
It's a bit like declaring your bike outperforms cars in crash safety because a magazine gives it 8/10 for handling and 9/10 for the visibility of your yellow jacket and your rear light having a "flash" mode.
Their FAQ just says "ask your insurance company" lol
As for the "who has liability" question it's simple: the driver does. Or at least it's intended to be simple in the absence of lawyers twisting the wording around. Regular, non self driving cars have basic driving assists. For example, cruise control will happily ram your car into the back of another one. Adaptive cruise still has a chance of doing that, plus it has the additional risk of causing an accident by randomly slamming on the brakes for no reason at all. For all these failure modes the liability is understood to lie with the driver: it is up to them to be alert enough to override the automation when it malfunctions. However, if you do a long distance drive with and without cruise control you'll quickly admit that it has value, even with all the imperfections. Same for Comma.
When you write software that can kill people, you don't get to roll your eyes at questions like "who is responsible when someone dies?".
These sorts of questions ARE engineering questions, and answering these questions with thought and care is important! Why? Because if the answer is "this is alpha quality and we might be liable" then you wait to deploy the feature. Which is why comma.ai is "ahead" of its competitors -- because they aren't doing real engineering. Thinking about the real-world context into which your system is being deployed is the thing that separates real engineering from R&D/hacking.
What you're describing is not engineering; it's R&D. Even comma.ai admits this. And, look, R&D is perfectly okay! Everyone else is also doing R&D on real roads! E.g., all of the major auto manufacturers are putting cars on real roads with full L5 (and safety driver backups where appropriate). In fact, the major auto manufacturers are all WAY ahead of comma.ai when it comes to R&D-quality systems! Compare Cruise or Argo or Uber ATG or Waymo to comma.ai.
But it's a terrible idea to ship R&D to paying customers when lives are on the line. If comma.ai wants to drive their system with their own safety drivers, that's fine.
This isn't even (just) a normative or ethical statement. It's simply a statement of fact, at least in the USA. For some reason Software never became a "real" engineering discipline. But automotive
Toyota was found legally responsible for its unintended acceleration bug from the mid 2000s when "the driver should pay attention 100% of the time" wasn't even something you would think to say because there were exactly zero driver assist features.
The driver is always responsible. This whole exercise feels like people trying to remove themselves from liability so that they can feel better about their absurd behavior.
Why can't we keep this simple? The driver is responsible.
When that didn’t work, I think they set out to develop a proof of concept that they could use to generate hype, then wait until an automaker wanted to buy their technology to catch up.
I get the impression they cursed future acquisition opportunities by refusing to even engage with safety regulators. They’re now selling their product in clever ways designed to shift all liability to the user.
In "clever ways" that completely fail to succeed in this goal, legally speaking. Product liability (law) is very expansive and generally covers every person in the chain from user to manufacturer.
On the one hand, if I get wasted and crash my car into a tree, insurers will generally cover that even though I was negligent. On the other hand, if I get cheaper home insurance by lying about having smoke detectors, and my house burns down, I might be in violation of the contract and not be covered.
So it seems like where this SHOULD end up is that insurers could charge more (or less as it gets better) for driving with these features. That said, I wouldn't fault an insurance company for not wanting to cover a system that says in all caps, as the linked article points out, "THIS IS ALPHA QUALITY SOFTWARE FOR RESEARCH PURPOSES ONLY." I'd possibly equate that warning to somebody who got into an accident after doing research on how well they could drive steering with their teeth, and it'd make me consider broadening the definition of "intentional" damage.
He makes this interesting observation that essentially all good drivers are alike; each bad driver is bad in his own way.
It's an interesting insight and great for self driving since the "good" drivers would be clustered while the "bad" drivers would essentially be noise.
He even mentions HN a couple of times:
I won’t be derisive as this author claims I would be, I’ll be serious. I’ve worked on autonomous driving solutions with traditional car companies, they move slower because BMW means something and they know it’ll mean death if they fuck up. Comma AI haven’t learned from the industry they’re trying to replace. This is the continual edge that Silicon Valley has over the rest of the world - they’re happy to just use legal arbitrage to gain an advantage and then pivot once their advantage is entrenched. It’s thus ridiculous “how were we supposed to know this would happen” excuse that’s trotted out all the time that’s absolute self serving bullshit.
sure self driving cars is the engineering end goal. But really we aren't we trying to cut fatalities?
Good driver monitoring is a pretty god Dayum good way to prevent distracted driving from causing an accident.