Openpilot – Operating system for robotics
github.com
github.com
I have a Comma 3x in the truck and felt way more confident, alert and comfortable for the entire drive. OpenPilot/Sunnypilot/Frogpilot are not FSD, but they are hands off driving assistance. The 2020 Ram performs incredibly well. The latest driving models are very smooth as well, no ping-ponging and they handle passing and traffic extremely well.
A legacy car maker would be smart to acquire Comma if its for sale. They would be extremely close to a viable assisted driving capability with it.
All Hyundai models in Europe have LKA, some (more expensive) also have LFS.
I unfortunately don't have radar cruise control on my Kia, though, which would make highway driving even in traffic completely effortless, and this seems to be standard on themore expensive cars. Maybe it's for the better, though, because it does force me to be much more attentive on the road.
It did some weird things like if the car in front of you was driving a bit too far to the left/right of a lane, it would copy them. Other than that it was nearly perfect, though. Never had it take an exit by accident, etc.
Their tuning on when to accelerate/brake and make it smooth needed a fair bit of work, but I found that switching the drive mode from Dynamic (Sport) to Comfort changed the eagerness of the system and smoothed things out.
Wouldn't that conceptually be the right thing for the software to do, copy the human in front of it (unless it has demonstrably better information)? OT1H, "lemmings," but OTOH unless the whole line of cars were all on openpilot my life experience has been that the person in front of me by definition has more visibility than I do since their car is not blocking their view as it is mine
I am totally talking out of school, because I'm not in that space and my poor BMW chose to do its own thing[1] so it doesn't work with openpilot[2] -- although they have a dedicated #flexray channel[3] so hope springs eternal
1: https://en.wikipedia.org/wiki/FlexRay
2: https://github.com/commaai/openpilot/issues/44#issuecomment-...
3: https://discord.com/channels/469524606043160576/533838492443...
I see people failing to follow the rules for bad reasons far more often than for any good reason. I don't want my car driving off to the side of the lane just because the car in front isn't centred. It should assume the right thing to do is to follow the rules, and hand off to me in cases that are more complicated.
Obviously a model which manage these conditions would fair better but the comma hardware is fairly underpowered for any stronger use case.
I have added dedicated compute to my car to handle a lot of driving rules but now my solution is independent of comma. I tie into the LVDS display on the console so the integration is immersive, but it also means I don’t need comma for the hardware. My fork is also starting to diverge from OP so I may have a competing (but tangential) product!
I run my own forked copy of openpilot and the car cannot keep up with turns in dynamic mode. When set to comfort it can handle all turns with ease.
My impression is that the Comma guys were never in this to sell their business
Obviously, FSD is way ahead of e2e open pilot with navigation, but since Open Pilot can apply very little torque to the wheel, it can't do anything gnarly. I actually trust Open pilot more at this point but I guess I just need more time with FSD. Some of that is because longitude was Toyota controlled until I used the e2e longitude model more.
Even on "chill" mode, FSD will make random quick lane changes to turn only lanes to try to get around traffic. This is 12.5.2. Even so FSD can get me from point A to B with no interventions 98% of the time.
Edit. I'm dumb. It's listed under "Ram" not "Dodge"
Also, most cars that have distance assist and lane keeping probably have the required hardware to control speed and steering to some extent.
Nevertheless, it's still impressive that so many cars are supported... and that it can be retrofitted like this at all!
https://www.reddit.com/r/Comma_ai/comments/197k04q/2023_kia_...
https://github.com/commaai/openpilot/issues/30936
You can find a few attempts of people trying to get it to work in their discord with no clear positive outcome, discord is unfortunately not search indexed.
I found out I needed to update some files in the firmware, followed the guide for that. Asked for help in the Discord. I could never get it working though and returned it after a week. I was going to hang on to it and harass people on Discord, but didn't want to lose track of time and go beyond the return window. Believe me, I really wanted it to work.
Looks like it was added to the supported list before the explicit support I guess
https://github.com/commaai/openpilot/commit/e3275e918354945d...
Using discord, etc for any open source project discussions is really unproductive.
Get it that people like the immediacy of chat style communication. But it seems to encourage way more noise and less thoughtful dialogue. The worst aspect is that it locks away a lot of useful information. Trying to dig through chat histories for information is a horrible experience.
But compare it to something like Microsoft support forums and you immediately realize why people are using it. It might not be the best implementation of chat, but the immediacy and lack of formality are undeniably valuable.
https://comma.ai/vehicles has also been improved quite a bit in the past year.
It's important to note that nothing we're talking about here is actually "self driving" per SAE standards. Openpilot, Tesla's Autopilot/FSD, Honda Sensing, Toyota Safety Sense, Hyundai SmartSense, etc are all level 2 driving assistance features.
This turns level 2 driver assistance features into ... nicer level 2 driver assistance features.
https://www.sae.org/blog/sae-j3016-update
Vehicles with levels 0-2 of driver support are never self-driving at any moment of operation. These systems might be able to manipulate the controls of the vehicle, but they are not good enough to make decisions without constant supervision at all times.
Level 3 is partial self-driving, where the vehicle can assume responsibility for driving under some narrow circumstances.
Full self driving doesn't happen until level 4.
I think the reason that there's no decimal points is because J3016 is really a road-map of the milestones on the way to full automation, not a buyers guide.
Things don't really get that simple until level 5.
Even at level 4, we're going to have vehicles that pull over when they can't figure out how to navigate successfully.
Minimal VC funding, less than 100 employees, not outrageously increasing headcount each month, profitable and sells a product with good margins.
Not many startups do this anymore, they are just chasing funding every 3 months using OpenAI’s API, comma has their own models before the AI hype.
Make business chill.
That said: The larger they get, the more regulator attention they’ll attract. If some government entity wanted to, they could probably easily kill them.
https://github.com/commaai/openpilot/blob/v0.9.7/LICENSE
https://github.com/commaai/panda/blob/32eecd721129b9215030c5...
Also, while poking around, I saw they use a "CTF" for on-boarding new contributors, which I thought was neat: https://github.com/commaai/openpilot/blob/v0.9.7/tools/CTF.m...
"THIS IS ALPHA QUALITY SOFTWARE FOR RESEARCH PURPOSES ONLY. THIS IS NOT A PRODUCT. YOU ARE RESPONSIBLE FOR COMPLYING WITH LOCAL LAWS AND REGULATIONS. NO WARRANTY EXPRESSED OR IMPLIED."
Where can this be used? In a private parking lot?
With the disclaimer that this depends on the location of course. For example, I think in Spain (and probably EU wide?) modifications that affect steering and throttle control would need to undergo local homologation before you're legally allowed to drive with that on public roads at all.
Which, to be honest, makes a lot of sense. I don't think anyone would be happy if cars start using software MVPs automatically controlling throttle and steering while in real traffic.
If you’re so confident this will drive my car into a ditch, then front the money for me to buy a compatible car & this kit. If it drives my car into a ditch, I’ll pay you back double that money.
That’s an easy bet for you, right?
Would you buy an angle grinder that's specifically been designed to not have a guard so it can be more useful, made by random contributors on the internet without any official certification? I'll stick with the ones that needed to pass the CE mark and you can sue the company responsible if it chops your arm off thank you very much. They at least have a required level of anxiety needed to patch anything serious knowing the level of responsibility they carry and what's coming if they mess up.
Suppose I deem it safe and useful for my 6yo to drive for a while, using his Xbox controller from the passenger seat.
It is illegal in many countries for a device (or anything else) to obscure any part of the driver's forward view (area swept by wipers). So even without actually controlling the car, we have an unlawful vehicle.
They say it's required for ISO compliance with Level 2 autonomous driving
They decided it wasn't worth explaining that their techniques don't generalise to a driver assist. It would not be good or cheap enough to be worth developing the compliance and integration frameworks.
It’s also possible they shifted direction because the long term vision of robotaxis is much more lucrative.
It is materially less safe to operate a ADAS while distracted than driving manually. Humans are exceptionally good drivers on average, only encountering minor crashes on timeframes measured in years to decades. As such, if safety critical ADAS errors occur more frequently than every ~100,000 miles and you are attentive in less than 100% of all such occurrences, you are operating your vehicle multiple times more dangerously than the average driver (which is a number that includes drunks and distracted drivers).
That is why it is critical to deliberately downplay the capabilities, to avoid wishful over-reliance, and enforce strict driver awareness (through techniques such as driver monitoring) to avoid operating multi-ton killing machines in ways that are multiple times more dangerous to the occupants, other drivers, and pedestrians. Without that, people are prone to over-generalization of safety capabilities, extrapolating that a single success means robust, continued success thousands to tens of thousands of times in a row.
I just don't believe the approaches for high autonomy (especially at the time) actually could make a cost effective assist system.
And for whatever reason they decided to push the we didn't like the driver behaviour message, rather than actually talking about what was actually plausible to achieve in the driver assist space.
I think this mode is something car manufacturers should enable in general. I actually suspect it's significantly safer than completely hands- and feet-free driving modes, and you get most of the benefit of lane-keeping assist.
1. Is this street legal? If so, how?
2. They discuss functional safety, and lots of testing, which is great. But I’d want to see some data on the test results — maybe this exists and I just didn’t see it?
3. It makes me uncomfortable that the anecdotal videos are easily findable, but bulk data/statistics are not. Anecdotes can easily be cherry picked. I get that they’re necessary for marketing, but I don’t feel they’re sufficient given the product’s purpose
I have 15k miles on it. Was able to retrofit a friend's 2015 car as well with a bit of additional hardware, and he likes it. He also has FSD on a model3. But OP or FSD, driver always has to pay attention and add their inputs.
The “horse winning race” case is a known one where they go into this.
No, it's not an OS for robotics. You can't do actual robotics stuff with it, like drive actuators to control limbs or grippers, do motion control or SLAM or perception or any of the usual robotics stack.
Their website correctly says openpilot is an open source advanced driver assistance system that works on 275+ car models of Toyota, Hyundai, Honda, and many other brands. Should've stuck to that.
Thinking about it some more, it's probably just another engagement baiting strategy to get attention and I'm their gullible puppet. Well played.
Took the bait as well.
The other half of the joke is that ROS was never an operating system either.
But the rest are firmly downgrades all around. It's slower (rclpy is catastrophically bad), more demanding (CPU usage is through the roof to do DDS packet conversions), less reliable (the RMWs are a mess), less compatible (armhf is kill). The QoS might count as an improvement for edge cases where you need UDP for point clouds, but what it mostly does on a day to day basis is create a shit ton of failure cases where there's QoS incompatibility between topics and things just refuse to connect. It's lot more hassle for no real gain.
And yeah I forgot, there's the added annoying bit where you can't build custom messages/services with python packages, only ament_cmake can do it so you often need metapackages for no practical reason. And the whole deal with the default build mode being "copy all" so you need to rebuild every single time if you don't symlink, and even that often doesn't work. The defaults are all around impressively terrible, adding extra pitfalls in places where there were none in ROS 1.
You're right ROS2 isn't all round better than ROS so the transition will never happen fully.
FWIW I'm working on an actual replacement for ROS, I'll post it to ShowHN one day soonish :P
So the claim still stands?
While I agree operating system is usually a marketing term, it does feel correct in this case as it is the operating system for the Comma Three, which can operate cars but also this thing: https://www.comma.ai/shop/body
> You can't do actual robotics stuff with it, like drive actuators to control limbs or grippers, do motion control or SLAM or perception or any of the usual robotics stack.
A lot of the "usual robotics stack" is not going to be relevant for the new wave of consumer robotics that is coming soon. It will be enabled by end-to-end machine learning and stuff like traditional SLAM methods will not be a part of that. The Bitter Lesson[2] is coming for robotics.
[1] https://x.com/__tinygrad__/status/1834792473081815259
[2] For those unfamiliar: http://www.incompleteideas.net/IncIdeas/BitterLesson.html
And well some classical algorithms like A* are mathematically optimal. You literally cannot train a more efficient DNN if your problem needs grid search. It will just waste more compute for the same result.
Besides, the nav stack is not really the point of ROS. It's the standardization. Standard IPC, types, messages, package building, deployment, etc. Interoperability where you can grab literally any sensor or actuator known to man and a driver will already exist and output/require the data in the exact format you need/have, standard visualizers and controllers to plug into the mix and debug. This is something we'll need as long as new hardware keeps getting built even if the rest of your process is end to end. It doesn't have to be the best, it just needs to work and it needs to be widely used for the concept to make sense.
And no, the usual robotics stack is not going away anytime soon. Maybe develop some actual useful robots before posting like an expert on robotics topics.
openpilot is an open source driver assistance system.
So what, you want everything written in RUST on a linux kernel with hard real-time patches? It uses machine vision anyway, which has no hard guarantees at all. The software it uses to detect lanes or cars is probabilistic by it's very nature.
Python does pretty good at soft real time if you manage your own event loop and disable the garbage collector, and you're a lot less likely to get "crash the entire stack" style memory allocation bugs. Sure, GO or RUST would be better, I think CPP could be worse if handled inexpertly.
I've been using Linux/BSD for over a decade now. No C or C++ application has ever crashed, I cannot say the same about Python applications. Outright segfaults are rare but happen. Rogue exceptions are much more common and could basically have the same detrimental effect on a self-driving system as a segfault. And let's not talk about logic bugs due to version incompatibility and the obsessive rewriting of those who took control over CPython.
Fair enough, your experience may vary. I'd suggest not judging the language by the standards of some hobbyist code that just so happened to end up on github. I've had tons of bugs in c/cpp programs over the years, some more critical than others.
I've seen a lot of shitty and unreliable python code, and a lot of good and mature C/CPP projects. I've also seen really bad security issues and crashes with bad C code, heartbleed, crowdstrike, etc.
For what it's worth I've never had youtube-dl hard crash on me, and I could argue that it's a more complicated problem to solve than what curl is solving. In an apples-to-apples comparison I think it does pretty well.
No matter what language you use for this you're going to be relying on an AI vision model with no hard guarantees.
except:
print(“Something happened”, i)
Fortran is able to generate better bugs, because it has allocate/free.
That said, I'm pretty sure CPython has exploits, too. They'll be harder to find and trigger though.
The first rule of the tautology club is the first rule of the tautology club. Things have trade-offs. Python removes (or at least significantly reduces) a whole class of bugs that appear when using lower-level languages, that's part of why it's a good glue language.
They also run pretty extensive tests (regression, unit, hardware/software-in-the-loop, mutation, and vehicle specific) on every commit and have actual hardware devices continually running real routes looking for regressions.
https://github.com/commaai/openpilot?tab=readme-ov-file#safe...
https://github.com/commaai/panda?tab=readme-ov-file#code-rig...
> a standalone device (the panda) that provides and enforces the safety model
What the actual safety model is that is being enforced is far more important here. The safety model could be "there is no safety guarantee whatsoever" and this sentence would still be true.
> All code is written in C to automotive safety standards including ISO26262, ISO11270, ISO15622, and MISRA-C.
26262 says practically nothing about software, what you really want is 21448. And 11270 and 15622 are super low targets for the amount of control authority available here.
MISRA-C is mostly a waste of time when it comes to safety. It gives software developers the warm blanket of having a checklist they can tick items off of, but does little to prevent unsafe systems from being built. Programmers have gotten pretty good about at least using tests and other analysis tools to make sure they're not doing the wildly stupid things that MISRA tries to prevent.
> 100% line coverage for all safety unt tests
100% like coverage is also rather trivial to achieve and doesn't say much. Branch coverage would be better, but being able to make some claims about state space coverage with exposure numbers would be what I'm expecting here.
yes in the sense that python is running the ML models and deciding what the vehicle should do, but it is heavily bounded in what it can do by the safety model which is implemented in bare-metal MISRA C running on the microcontroller that interfaces between openpilot and the CAN bus (panda). It enforces things like accel/braking limits and steering rate limits along with consistency checks, heartbeats, vehicle status checks, etc.
Level 2 self driving is already only a best effort system so if python caused an issue it would just fall back to the safety model on the panda and ultimately the driver to operate the vehicle safely.
Sure you have to actively be alert your entire drive, but it’s still significantly better than actually doing the work of driving.