California halts Pony.ai's driverless testing permit after accident
reuters.com
reuters.com
"On October 28, 2021, after turning right onto Fremont Blvd from Cushing Pkwy, the Pony.ai Autonomous Vehicle ("Pony.ai AV") performed a left lane change maneuver in autonomous mode. While performing the lane change, the Pony.ai AV came into contact with a center divider on Fremont Blvd. and the traffic sign that was posted on the divider. The Pony.ai AV suffered moderate damage to the front of the vehicle and the undercarriage. There were no injuries and no other vehicles involved. Fremont Police Department were called to report the incident and the damaged street sign. Pony.ai has subsequently worked with local authorities to resolve all issues related to the damaged sign."
It's not a serious incident, but it is a fully autonomous vehicle under good conditions going somewhere it was absolutely not supposed to go. So DMV revoked their license for full autonomous operation without a safety driver. They have to go back to testing with a safety driver.
https://www.google.com/maps/@37.4905025,-121.9488087,3a,24.2...
You can see the curb rashed up and the pretty improvised fix to the sign from a previous accident I suppose, since the map says March, 2021.
Using invisible-to-humans paint on signs for adversarial recognition attacks on autonomous driving systems would also be an anti-Tesla tactic not entirely unreasonable to consider.
"Tesla autopilot crashes right outside Tesla factory!" would be a desired headline for a certain crowd.
It'll be interesting to see what caused this, at any rate, probably less than 1% chance the above is what happened.
https://www.google.com/maps/@37.4902551,-121.9483919,3a,75y,...
The pole at the end of the divider with the sign facing away, stretched-arrow street paint left-turn symbol, and the enormous expanse of intersection pavement are all a little disorienting.
1. Don't hit anyone
2. Don't hit anything
3. If forced to violate rule(s) above, preserve the life of the person making the car payments
The most basic one, if indeed the outside is unknown to your product, are you a moral person developing it.
Sounds like a continuation of the automotive trend that pretends like the only person whose life matters is the occupant.
And perhaps that's part of the problem. "Oh, this person crashed his heavy machine into something...it happens" seems to significantly undervalue the responsibility and risks of driving. It should really be a "never" problem for people, too.
This is really bad, if their software can't even avoid hitting a stationary object then it isn't ready to test on public roads.
I know nothing about nothing, so it's a real question.
Their full self driving beta requires constant vigilance because it likes to randomly veer at pedestrians and light poles.
As a sidenote, it's irritating that Reuters keeps calling it an "accident." It was a crash.
Is this supposed to be a bad thing? Why?
The focus on the price of the house seems pretty strange in an area with a particularly high price-to-rent ratio. But I guess $20k/mo just doesn’t sound as spicy.
Otherwise, spending investor money on office space seems like a perfectly normal thing to do. The $ amount described doesn’t seem too unusual for the area, and a house might offer significant advantages over traditional office space for a startup that is working on cars.
The price mentioned isn’t anything unusual even for fairly early stage startups in the Bay Area.
I see you have something like a half dozen comments now trying to rationalize this all cross the comment section, but like yeah renting out a mansion for your startup does send out a "vibe".
If you don't realize that a) you're probably who they're looking for, and b) you probably have some room to grow when it comes to picking up on subtext.
-
Not everything in life can be quantified in a neat little bullet pointed list. But part of being an experienced person is being able to pick out the more subtle aspects of situations.
To me a mansion as a startup HQ screams tech-bro 24-7 party culture with some major WLB issues.
And again, it's not like using a mansion magically causes that means that, but it strongly implies it.
It's like walking into an office and seeing everyone in suits and ties vs seeing everyone in sweatpants and baggy tees.
Technically the choice of clothes does not directly force you to behave a certain way, but your intuition should tell you that those are two very different cultures, and it should tell you a little bit about each.
tl;dr Ignoring intuition because it's not backed by unequivocal fact is not a super-power, and it's not really productive.
Don’t get me wrong, I’d never work with the manchildren at comma.ai. I’ve interacted extensively with George Hotz in the past and he’s certainly not a person I’d want to spend any time in the same room with.
I just don’t see any issue with their choice of office space, it just seems like a perfect fit for their business.
> To me a mansion as a startup HQ screams tech-bro 24-7 party culture with some major WLB issues
Where else would you put your car tech startup in SF? Software engineers aren’t going to work out of a garage, and good luck finding a traditional office space with decent facilities for working on cars.
FWIW this is hardly a mansion https://www.redfin.com/CA/San-Francisco/41-Santa-Monica-Way-...
Comma.ai has a “proper” facility now, but they had to move off to San Diego for that.
At the same time as the $10m mansion place I interviewed for a self driving truck company and they had their office located in an industrial park in a shop that looks like it used to be an auto mechanic with upstairs offices. That place was fine but I didn’t feel like working for a 22 year old CEO who had just left college. Back in 2016 investors were dumping money on self driving car startups but I wanted something stable.
In…a garage. I’m serious - there are spaces big enough in SOMA and other places that can fit a garage space plus open office space plus some conference rooms (I’ve worked in several of them).
I had just gotten out of a startup where the guy treated the place like his personal clubhouse and had a really controlling and abusive attitude. I wanted to steer clear of any company that seemed inexperienced.
It apparently doesn’t look that way to you, but that’s how it looked to me and I see other people here can relate.
* Are the founders or employees living in their office?
* Are they treating it like a work environment or a party house? Are the work, kitchen, and bathroom environments professionally maintained.
* Is there adequate parking or public transportation?
* Is it legal? Does it conform to HOA and city requirements?
* Is it centrally located for commuting workers and testing sites or the personal preferences of the founders?
* Is it scalable for a growing company?
Renting a mansion as an office throws a hundred red flags to anyone wanting a serious outcome from a startup.
Wasn't snap for example running out of a few houses on the la beach? Maybe they weren't mansions, but I'm sure the rent was high.
> Is it centrally located for commuting workers and testing sites or the personal preferences of the founders?
The house in question was probably the comma.ai office in West Portal, SF (right next to the West Portal Station). Draw your own conclusions regarding commutes, it’s a pretty central location.
> Is it scalable for a growing company?
Offices are rarely very scalable, you just move somewhere else once it no longer works.
The point of my original comment is not to say that doing so is objectively bad, but I had just left an abusive startup run by a young and inexperienced startup and I wanted something more stable. I ended up getting a job at a Toyota research branch in a boring office park and I played with a cute little robot. It was just what I needed at the time.
This is common for startups in the seed stage. I don't personally recommend it but I wouldn't consider it to be a red flag for a pre-series-A startup.
> * Is there adequate parking or public transportation?
There is practically no public transportation in the bay area. No, there is never enough parking.
> * Is it legal? Does it conform to HOA and city requirements?
Nobody takes a shit about HOAs. Startups exist to drill square pegs into circular holes. Hell, I'd love to create a startup to help people get rid of HOAs.
I had also just left a startup where the CEO treated the company like some boys clubhouse and I did not want to get in to a situation like that again.
Pretty sure in 2016 Comma.ai was at 41 Santa Monica Way in West Portal.
Just slap a Beta tag on it and she'll be right.
Almost as bad as a Tesla crashing into a stopped fire truck. Why is Tesla “autopilot” still allowed on Californian roads?
https://www.wired.com/story/tesla-autopilot-why-crash-radar/
PS Tesla rolled a software update to detect emergency vehicles and slow down.
https://www.reuters.com/article/us-tesla-autopilot-germany/g...
Tesla's Full Self Driving has been released only to a tiny number of beta testers.
2) Yes, and the videos I’ve seen from those beta testers are terrifying.
Even taking the lesser of these two scenarios, this would imply that Tesla autopilot should be capable of doing most of the driving, with the driver just there for whatever bits you might argue are equivalent to a takeoff and landing. Given the stories of Teslas accelerating into stationary objects or totally losing track of the road lane, I don’t think even this limited understanding of what “autopilot” means is correct.
> "The person in the driver's seat is only there for legal reasons. He is not doing anything. The car is driving itself."
> “Full Self-Driving Capability All new Tesla cars have the hardware needed in the future for full self-driving in almost all circumstances. The system is designed to be able to conduct short and long distance trips with no action required by the person in the driver’s seat. All you will need to do is get in and tell your car where to go. If you don’t say anything, the car will look at your calendar and take you there as the assumed destination or just home if nothing is on the calendar. Your Tesla will figure out the optimal route, navigate urban streets (even without lane markings), manage complex intersections with traffic lights, stop signs and roundabouts, and handle densely packed freeways with cars moving at high speed. When you arrive at your destination, simply step out at the entrance and your car will enter park seek mode, automatically search for a spot and park itself. A tap on your phone summons it back to you. The future use of these features without supervision is dependent on achieving reliability far in excess of human drivers as demonstrated by billions of miles of experience, as well as regulatory approval, which may take longer in some jurisdictions. As these self-driving capabilities are introduced, your car will be continuously upgraded through over-the-air software updates.”
That’s also why most TACC doesn’t work below 30km/h or so. It can’t tell the difference between the car in front and a stationary object next to the road.
Edit: This is actually an interesting rabbit hole to go down, because on the face of it, it seems like a fairly straightforward system to implement. Then the more you think about how a 'simple' system should handle various scenarios, the more you realise that to actually do it properly, 'just keep a good distance from the car in front' requires most of the perception required for general free-form driving.
https://youtu.be/0A4i4-xALVg?t=252
Edit;
I guess you did say braking with TACC is complicated -- but I think that's a bit of an edge case that only applies to Tesla. The Subaru version of TACC is ~fine but it never gives you the impression it will handle 100% of the driving for you, so you'd never leave it to navigate around emergency vehicles by itself. Maybe the solution to rarely dangerous self-driving is to make them only ~75% good so driver's are always aware?
Do you have one (or experience driving one)? If so:
- How does it go with phantom braking and various objects on the side of the road?
- Does it distinguish between various on-road things? (eg. does it brake for a dog? a plastic bag? a puff of smoke? an anvil falling off the back of the truck in front?)
- How good's its understanding of lanes (eg. a very slow car that's in another lane but currently directly in front of you due to the road curving)?
- Does it brake for objects outside the lane entering it (cross traffic, lane cut-ins, pedestrians, vegetation moving around in high winds)?
You're still responsible for watching traffic and using the brake or gas as needed, especially if there is some issue up ahead that requires a safety maneuver of any kind.
i.e. an emergency vehicle, like a California Highway Patrol vehicle, begins a traffic break 15 car lengths ahead of you. As long as you're observant, you'll notice that in time and either cancel the cruise control or make the right adjustment so you slow down enough to the break speed.
Citation needed. Subaru claims their eye sight system will detect and attempt to brake for a stationary vehicle. Whether or not they’ll managed to halt the vehicle is obviously speed dependent, but obviously slowing down before the crash is vastly superior to maintaining speed.
When the vehicle is stationary, the radar data has to be heavily filtered because radar has poor angular resolution and there are a lot of other stationary objects, many that you don't want to stop for at similar angles, like overhead road signs. That means you're relying purely on vision, which alone has a lot of false positives and negatives, so detection thresholds need to be set much lower.
Doesn't mean it won't work. Just means it's less likely to work, especially in the dark and the rain, where radar would normally be a big part of the signal.
https://www.engadget.com/2018-03-24-tesla-model-x-driver-die...
The driver of the tesla had complained in the past of the tesla swerving into the wall when passing that spot, yet still had his guard down on the day he died.
That the tesla swerved is down to the poor striping on the road. The stripe along the right side of the left fork was brand new and very sharp, while the stripe along the left side of the right fork was extremely dim to non existant due to excessive wear sustained during construction of the left fork and the bridge it leads to (connection from the fast lane of 101s to the fast lane of 85s).
The crash barrier has since been properly replaced and the striping redone with diagonal lines hashing off the gore so it's abundantly clear even to a camera not to drive there.
That said, the tesla self driving made a very dumb error no human driver would make.
No, it's down to defective self driving.
Certainly not. It's not appropriate for a self driving car to make any assumptions about striping or any road markings being correct. It crashed because it wasn't capable of self-driving after all.
Because Pony.ai is level 4 autonomy while Autopilot is level 2. The human driver is ultimately responsible for any accidents in a level 2 system. There is no driver to intercede or to blame in the case of a level 4 system. This crash is the sole responsibility of Pony.ai.
Because Tesla suckered some idiots to take the blame if for example the car will instantly change direction and throw you of a bridge or in from of a truck, Tesla is saying that their customers are supposed to be ready for this cases .
For now only some Tesla alpha testers suckers got killed and no innocent person, but when an innocent will be killed I am waiting to see if the driver and hist lawyer will accept the blame or will fight the ToS claims.
People are bad drivers because they have limited vision, limited attention, are emotional and are easily distracted by all kinds of things.
Just because an autonomous car has a similarly bad track record as the average human, doesn't mean it's ready for the road.
Of course, if that human brain is tired, stressed, and checking their phone, it's a whole different story :)
So I feel like no matter how many autonomous hours these cars accumulate on the road, they have effectively 0 hours of "learning to drive".
I doubt it, not anything serious at least. Machine learning isn't at the level where it can be used to navigate environments better than human coded algorithms can. Maybe if you had a datacenter worth of computing power to do it, like AlphaStar, but you don't have that in a car. And even if you had that I still doubt it, Alphastar had a perfect representation of the game and could simulate the effects of moves perfectly.
A driving AI has no general knowledge. It doesn’t actually understand what it is looking at.
It’s not possible to make up reasonable responses consistently when you don’t know what you are looking at most of the time, machine or human. Have you ever tried to solve a technical problem before you read the documentation covering the basic concepts? You’re basically throwing shit at the wall until something sticks.
Full disclosure, I work for Aurora in the Planning group.
Edit: To clarify, I wonder how you do this part:
> make safe, predictable decisions on the road.
ML models doesn't make predictable outputs. So you always need a system that can handle the ML model going haywire, since you have no way to prove that it wont.
As for stack accuracy and predictability, they're active areas of research for everyone, and part of the secret sauce. There's lots of little mitigations scattered throughout everyone's stacks for various particular issues, but no silver bullet.
So a 100 hour old infant?
Here is a 50 hour old horse, imagine if our self driving cars had this level of environmental awareness:
Biological brains are super cheap and efficient though. The brain of a fly can still do object recognition and navigate environments for basically free, weighs less than a milligram including cameras, and the factory to produce those brains is just putting a cell in a wet environment and it builds itself, the brain of an ant can build complex structures etc. It is possible metal brains will never beat those.
(Amusingly, while standing up is clearly built in - foals usually stand on the first try - lying down is not. I've seen a newborn foal try to lie down for the first time, and after moving a few legs, just collapsing to the ground. This makes sense - there's a survival advantage to being able to stand up quickly and reliably, while being able to lie down and rest quickly need not work as well.
Animals have a heck of a lot of background knowledge like that. Statistical machine learning algorithms can't incorporate background knowledge [edit: not easily] and have to learn everything end-to-end. Hence why they remain dumb as bricks, compared to even simple animals (like insects, which are "simple" only compared to mammals, say).
Unless you count things like
https://en.wikipedia.org/wiki/Out_Run
The car controls are not the challenge. It’s understanding what everyone else is doing and how to handle it safely while making progress and not surprising other people that is tough. And those skills are partially - only partially - transferable from your pre-car experience.
A lot of that has to do with humans not being born with fully developed brains. Animals can walk and navigate environments as soon as they are born, so there is no reason to believe that humans actually learns these things rather than those systems slowly maturing as we grow up.
The other part is that our pre-trained movement system is made for human bodies, not cars. So learning to drive a car is learning to use another mode of moving yourself. Also you have to learn traffic rules. But the other things like understanding environments etc you get for free for being a human, every single large animal can do the same. They wont learn the traffic laws or how to drive the car, but they know how to navigate environments without hitting things.
Things in front are highly compressed, while things to the sides are not, and your ability to move sideways is reduced by mass and speed.
When I was a teenager, I went on a road trip where I drove like 12 or more hours in a day, which I could never do now. At the end, I got home and for a while I had a weird sort of tunnel vision that I don't know how to describe. It was almost like I was looking through a fish-eye lens or something; everything seemed distorted because of spending so much time concentrating on small lateral motions and things coming towards me at highway speeds.
Sometimes I walk routes that I also frequently drive and it reminds me how motor vehicles compress time and space.
I'm doubtful that current ML techniques are that general.
There are young kids that drive carts and race and they don't confuse walls and solid objects. The reason we don't allow younger people to drive is because young people are bad at managing risk(I done some very stupid risky things with my bike as a teen, I could have broke my bones or neck),
Autopilot from Tesla that uses mostly or only cameras has the issue of identifying obstacles which is a much generic problem(that young animals and kids have it solved) so you don't need to appeal to the "age of driving" excuse for this guys.
So many people don't even know all the rules of the road, but still manage to drive just fine without accidents. There are different unknown rules of the road in all different places around the world, but most people figure out how to adapt to them in a very short time.
There are huge differences in insurance premiums that can't be completely unrelated to accidents. People like to talk about other factors, but still.
Do you think we live in StarTrek future already? Human brain is the most complex thing in the entire universe, as far as we know so far. You compare it to a what, a computer?
Because that's the only motivation for keeping things safe, if it's just a fine corporations will just cut a check and write it off and shrug.
Cyclist on side of road, pedestrian wearing the wrong thing crossing the road, etc.
Sure a human could have killed them too. But that human will be put on trial/jail.
Unless the programmer intends it anyway, then they are the murderer.
Same as a lion can’t murder a human. It’s just being a lion, doing lion things.
((I had to.))
The end result is that Uber ended their self driving car program and paid the family money, while the grunt worker who was put in the car without proper training got charged with negligent homicide and that trial is still going.
The person responsible for that program should be charged, not the safety driver, but that is how things goes. You can't just put random people in a car and call them safety drivers, and charging such a person with homicide doesn't really get to the root of the problem.
In reality there were 100s of people in the chain of events, from the safety driver being bored and watching their phone, to the person who hired them and didn't train them properly, to the person who invented the safety driver system and didn't think people would get bored.
Then you have the people who made the decision to turn off some of the subsystems of the car and then test it at night with no extra precautions, and every programmer who let that happen.
The the management of the company who provided the wrong incentives to their employees.
And that's before you get to the people who designed the roads, and the lack of social care to look after people who for some reason are homeless
The problem is it's not really that person's fault either if they were pressured by executives to have a demo in 2 weeks for some investors.
If the investors were of the "demo or die" mindset then the investors bear part of the fault as well.
There's no way it's ready for the sort of adversarial stuff you'd get hiring a bunch of people to drive around simulating traffic for a car. I guarantee it's a great way to find out how your car is at dealing with being cut off, at people running stop signs, etc - and there's certainly value in doing that.
Just, not while you still can't avoid hitting stationary objects along the edge of the road.
Some self-driving cars can't.
At one end, you have the Uber self-driving vehicles which were caught committing numerous traffic offenses by dashcams and cyclist helmet cams within the first days of their illegally testing their fleet on public roads, and then one of their cars ran over a pedestrian that the vehicle's sensors detected but the computers misclassified, somehow - and Uber had disabled the SUV's own auto-braking systems that probably would have saved the woman's life.
At the other end you have Google's self-driving cars have had, I think one at-fault crash across the entire fleet in many, many years and millions of miles of testing? They've been involved in a lot of crashes, but every single one until their one slow-speed scrape with a bus was the fault of another driver. That was 5 years ago or so.
Somewhere in-between is Tesla's FSD "Beta" which, as has been shown by numerous youtube videos (at least, the ones that Tesla doesn't abuse DMCA to take down), behaves extremely erratically repeatedly in the space of less than half an hour in an urban environment. This includes veering off a turn at pedestrians on a curb corner, and in another incident veering toward a light pole. In general, it seems to have a lot of problems making turns.
Basically stuff the owners don't do, but they're perfectly fine if it happens to others (for the pollution bit, I'm reasonably sure the stakeholders and executives live in nice, low pollution areas).
Full disclosure: Google employee, working on nothing to do with Waymo.
[0]: https://blog.waymo.com/2020/09/the-waymo-drivers-training-re... [1]: https://www.wired.com/story/google-waymo-self-driving-car-ca...
I think the car should also talk about what it thinks, to get people to understand the limitations of the technology.
"There could be a child between those parking cars, slowing down"
"I cannot see the road markings, emergency stop"
"Road signs don't match my data, slowing down"
"Train mode activated, following virtual track"
Then, and only then, you can do the second step. Automated cars have the potential to be much safer than cars with human drivers. You can accept some accidents, in order to prevent many more. But you can only do that if the public and lawmakers have confidence in the technology. Right now it is way too much of a black box with wierd failure modes.
The situation: The vehicle was on Vallejo crossing west to east over Sansome Street, a one-way, two-lane thoroughfare. For Sansome, there is no stop sign here, but there is for Vallejo. The vehicle stopped at its sign, then began to proceed across the intersection, until two pedestrians entered the crosswalk from south to north along the far side of Sansome. The vehicle stopped in the middle of the street to wait for the pedestrians to cross. There was no cross traffic coming, but what if there were? Seems like a situation ripe for a side collision, especially because Sansome is brisk at this point.
I really don't know what the capabilities of the sensors are, or how situations are assessed, so I'm assuming it did what it did based on its situational awareness. Wondered if the vehicle could have backed up to the stop sign it came from if there was cross traffic, or if it could have broken the rules and gone around the pedestrians if there was danger of a side collision.
The intersection: https://goo.gl/maps/TisMyYc1RhKgJWFf8
PS: there is a garage at that corner that almost exclusively contains Waymo vehicles, and this was not a Waymo car.
Computers, not so much.