Upgrading Autopilot: Seeing the World in Radar
tesla.com
tesla.com
This device isn't enough for a full point cloud. It doesn't scan in elevation, just azimuth. Some variants do have an upward-pointing beam in additional to the main forward beam, which is about 5 degrees in vertical.
There are automotive radars which scan in 3D[2], but Tesla's is not one of them.
Small radars are rather blunt instruments. You tend to get one point for each target, not lots of points. The beam focus isn't that tight. Tighter focus requires a larger antenna array.
[1] http://www.automotiveworld.com/news-releases/bosch-presents-... [2] http://www.fujitsu-ten.com/business/technicaljournal/pdf/38-...
Despite the hype, this blog post makes it clear that Autopilot is just a very simple camera based system that requires human monitoring (but is still very useful in my personal experience). Now they are releasing a software update that uses the previously unused radar hardware as well as machine learning to achieve better results. This is exciting because they're going to keep releasing software updates over the air and the software is just going to keep getting better and better.
The fact that the hardware is shitty doesn't concern me. Google's self driving car project depends on a Lidar system that costs $75,000 today. That's almost four times what the average car costs, and it doesn't work in bad weather. Regardless of how good Google's software is it could mean nothing if Tesla acheives a similar or better result using cheap commodity hardware. The hardware will only get better over time at the same price.
Tesla is in a better position to bring full autonomy to market than anyone else, since they control the hardware, the software, have cars in the field etc. For this reason I wouldn't be surprised if Tesla becomes the first company to break a market capitalization of $1 trillion dollars. Computers gaining the ability to move around the world with drones and autononomous driving will have an economic impact bigger than the introduction of the internet. Where we are now is meaningless. What matters is that Musk has stated the goal and we have something in the field today that can be updated iteratively over time until it's perfect (which will be never). We may not be very far along this journey, but we've taken those hardest first steps towards the next big technological revolution that will once again change everything about how humans live, the implications of which we can't even begin to imagine.
Sure, Autopilot is safe 'when used correctly,' but I wouldn't trust someone to maintain 100% attention on driving with it enabled. Maybe for a few cumulative hours. Not for hundreds or maybe thousands of hours. If I'm paying attention and I prevent Autopilot from getting me into an accident, why have it on in the first place? It's supposed to protect me from inattention! (e.g. automatic braking)
As it gets better at protecting people from inattention, people will be less attentive, and more will encounter the edge cases that the machine learning models will inevitably have.
Don't get me wrong, I love SDCs and the massive impact they will have, I just believe that partial autonomy is unsafe because of the human factor.
For every Tesla that is now able to recognise a truck cutting across a highway and brakes in time to avoid a fatal collision you will have new bizarre scenarios like following behind a truck that dislodges overhead objects, which a human would have avoided but a Tesla's computer didn't see, leading to a human getting crushed by a bridge falling down.
I would also expect a short learning curve for Teslas to acknowledge the emergency stop water curtain signs.
But the short version is yes, SDC will result in more deaths even though they get relatively safer and safer.
Because it takes less mental effort to watch for the exception case rather than constantly adjusting speed/heading/etc.
Also something that's rarely mentioned is that in it's current form AP's killer-feature is rush hour traffic. In those situations you're dealing with <20 MPH and tends to be the most mentally taxing.
In my small sample set(~30k AP miles) I've found the mental load for driving long distances and rush hour traffic to be significantly less.
Interesting---how did you measure this?
I've been doing the same ~7hr drive + workday for 2 years. Both myself and my wife noticed I was a lot less wiped out after AP. We have the car ~3 months before AP so it was a pretty controlled change.
Yes, this!
The argument that drivers are safer if they are forced to pay attention all the time is ridiculous, you can use that to argue against every feature on the car! Servo steering is bad, you don't feel the road properly! Automatic gearboxes are bad, you're not in tune with the engine! ABS brakes are bad, you should know how to brake so as not to lock the wheels! Traction control is bad, you should never drive on surfaces where you're not in 100% control of the car! Blind spot radars are bad, you should always look around you and make lane changes responsibly! Bla bla bla.
The current Level 2 autonomous systems makes it easier to do the right thing when driving. Used correctly, they lessen the cognitive load, but you have to learn how they work, what they can and cannot do, just like you need to know what situations your traction control systems or anti-lock brakes can save you from. They're not magic, they're just one more helper system.
The best human drivers can get by without the first set of features.
How is routing not human-enhancing? It makes it easier for you to choose which route to take? How is an automatic gearbox not human-replacing, it eliminates the need for a human to shift with the shift stick?
At ~30k miles, chances are you haven't seen enough exceptions to make a judgment on that.
It is hard to stay focused on jobs that are extremely dull, except for the rare cases where they aren't.
There's tons of research on this because there are many people in that situation (watching radar screens for incoming ICBM's, guarding a facility, luggage screening at an airport, etc)
Flying tires, wildlife jumping out. Heck, just yesterday had someone merge into the lane with no signal. Just because I'm not providing steering inputs doesn't mean that I'm not constantly scanning traffic.
The problem of Level 3 autonomy is not cognitive load but the general lack of active feedback on the one hand, and a lack surprising events to mediate the allocation of attention on the other. It makes sense, the longer you're encountering non-events, the more difficult it will be to justify alertness. Surprise guides attention which in turn is strongly correlated with alertness. The longer one must remain alert, especially without surprise, feedback or reward, the higher the levels of subjective effort and mental fatigue.
Intuitively, if you imagine that only a few things are filtered to attention, the less surprises, the more predictability, the more difficult it is to select what to attend to, the more likely your mind will begin to wander. This is one explanation for the finding in [1]:
The fact that the number of false alarms increased suggests
that the rather dramatic increase in missed targets was not
due to a simple reduction in the number of responses:
the number of responses to nontargets even increased.
This suggests that the observed deterioration of performance
is not caused by task disengagement but may result from increasing
difficulties for subjects to correctly identify targets
[1] In the linked article they specifically call out driving as an activity sensitive to mental fatigue http://www.sciencedirect.com/science/article/pii/S0926641005...
Last I heard, Autopilot was MUCH safer per mile.
The same reason you have cruise control - it doesn't do everything for you, but it helps reduce cognitive load in some basic areas, which reduces fatigue over the long haul.
Volvo uses four cameras (one of which is a trifocal 3D unit), four radars, one LIDAR, and 12 ultrasonic sensors. That's a reasonable sensor suite for this. Tesla has one camera, one radar, and some ultrasonic sensors. Not enough.
Volvo is also way ahead on self-driving car commercials.[2]
[1] http://www.volvocars.com/intl/about/our-innovation-brands/in... [2] https://www.youtube.com/watch?v=bJwKuWz_lkE
https://www.google.dk/amp/s/amp.theguardian.com/technology/2...
And better yet: their 100,000 (soon to be 500,000) autopilots will gather live information from roads from several countries, weather types, etc. even when the cars are manually driven. Every day, Tesla's 100,000 cars will bring as much data for their algorithms as Volvo's 100 cars do in 3 years.
Or put in another way: unless Volvo magically have much better programmers than the IT company Tesla, they would need 3 millenia to keep up with Tesla's current fleet of 100,000 data collectors.
Also, do you have any source for Volvo's current system collecting data and sending it to Volvo? We know Tesla does it but I did not know Volvo does it.
I don't know why, either, since this would presumably be very easily debunked.
For a representative example, see how bad Mercedes’ latest and greatest E-Class w/ Drive Pilot is: http://www.autofil.no/936897/hands-off
It’s damning. And that is the one they advertised as being “self-driving”. That’s the best Mercedes has.
This review compares more of them: http://www.caranddriver.com/features/semi-autonomous-cars-co...
The best of non-Teslas have at least twice as many autonomous mode disengagements (i.e. they suck at the lane keeping job). They specifically call out Model S for being the only one that can keep in a lane without wobbling like a drunk driver. None of the “old” car manufactures can even get that right, let alone anything more.
So yes, on paper all these components exist for 3+ years and are nothing new. Exactly like smartphones existed before the iPhone did — remember those?
[1] : https://www.media.volvocars.com/global/en-gb/media/pressrele... [2] : http://www.bloomberg.com/news/features/2016-08-18/uber-s-fir...
By the way I don't think Volvo is freaking out so much as going for it's vision 2020 thing:
"in 2008 we set out our vision that by 2020 nobody should be seriously injured or killed in a new Volvo car."
> Volvo Cars was owned by AB Volvo until 1999, when it was acquired by the Ford Motor Company as part of its Premier Automotive Group. Geely Holding Group then acquired Volvo Cars from Ford in 2010.[0]
> Geely (officially Zhejiang Geely Holding Group Co., Ltd) is a Chinese multinational automotive manufacturing company headquartered in Hangzhou, Zhejiang. Its principal products are automobiles, taxis, motorcycles, engines, and transmissions. It sells passenger cars under the Geely and Volvo brands and taxis under the London Taxi brand.[1]
[0]: https://en.wikipedia.org/wiki/Volvo_Cars [1]: https://en.wikipedia.org/wiki/Geely
That is silly overkill, and most of it is incompatible with visual cues.
I an human. I have only 2 cameras. I kick ass compared to these systems.
I can handle snow hiding the lines and edges of the road. I can handle snowbanks changing the shape of the road. I can handle cops waving and pointing. I can handle dirt roads. I can handle unmarked parking on grass. I can handle broken traffic lights, both individual and power outage. I can handle a total GPS outage. I can handle areas without cell coverage. I can handle new roads. I can handle traffic being diverted onto the opposite side of the road, or even off the road, as happens for construction. I can handle gaping unmarked sinkholes.
I believe you're multiple orders of magnitude off here. Google's LIDAR is surprisingly cheap. I believe they used expensive Velodyne units in earlier versions.
Heck, even brand-new Velodyne devices are only $8k.
They won't be able to achieve similar or better results than Google with the current sensor suite, for example they don't have anything to check for oncoming cars behind the vehicle. No software update can compensate for things that the sensors don't see.
Then how does a line changing feature works? I am pretty sure you need to see cars behind you to safely change lanes?
For safety reason the driver has yet to be sure that no one is approaching you at the moment of changing lane but the system obviously works autonomously, you just have to engage the turn signal.
> "Other manufacturers have rear facing radars for this."
Tesla have rear radars, otherwise Autolane-changing and Autopark features wouldn't be available to use.
https://www.engadget.com/2016/09/11/tesla-s-next-autopilot-u...
If Engadget got that right I think we will see a lot of upset Tesla owners in a few weeks.
Probably also a small handful fewer dead ones over the next few years.
ONE person died. Over many million miles of Autopilot driving. Last I heard it was much safer to drive with Autopilot than without. Yep, per Musk:
> “Indeed, if anyone bothered to do the math (obviously, you did not) they would realize that of the over 1M auto deaths per year worldwide, approximately half a million people would have been saved if the Tesla autopilot was universally available. Please, take 5 mins and do the bloody math before you write an article that misleads the public.”
There are luxury car models with a similar amount of purchases and miles driven that have had 0 casualties.
https://www.engadget.com/2010/02/11/south-korean-iphone-user...
edit: Upon closer reading, he explains it somewhat. Once they have enough data, the system will start braking on unknown objects with gradually increasing force as the confidence level rises. So basically, it will brake on unknown traffic signs but only slightly, as the confidence level shouldn't get too high, if I understand that correctly.
The last paragraph sounds technically challenging and interesting:
"Taking this one step further, a Tesla will also be able to bounce the radar signal under a vehicle in front - using the radar pulse signature and photon time of flight to distinguish the signal - and still brake even when trailing a car that is opaque to both vision and radar. The car in front might hit the UFO in dense fog, but the Tesla will not."
edit: it seems like they are already doing it beginning with this update: "Now controls for two cars ahead using radar echo, improving cut-out response and reaction time to otherwise-invisible heavy braking events". that sounds awesome.
blacklisting would cause the behaviour you describe.
So as i read it, the system does think it will collide but ignore it if the whitelist says the object is safe.
The paragraph after that says basically once they have enough data, the system will start braking on unknown objects with gradually increasing force as the confidence level rises. So basically, it will brake on unknown traffic signs but only slightly, as the confidence level shouldn't get too high.
Corner case: truck jackknifed across the road immediately under a white listed gantry.
Once enough human or autopilot trips detect the same signal with no collision, whitelist the signal and cease braking-and-monitoring behaviour.
I am probably misinterpreting but that is what makes sense to me.
But for the learning part, they probably combine the camera that they used to date as their primary device in combination with the radar (at least in daylight scenarios) to identify objects. They may even be able to learn about the special material properties, like the reflective coating of a traffic sign.
Because radar does not have the same resolution as LIDAR.
EDIT: Phased array radar and cheap stationary LIDAR should get under $100 in ~5 years, at which point this whole argument will be moot. Hacks in the meantime!
Could multiple radar emitters and receivers be used to create a phased array and improve resolution?
http://www.radartutorial.eu/01.basics/Range%20Resolution.en....
For angular resolution: http://www.radartutorial.eu/01.basics/Angular%20Resolution.e...
Depends on the radar system. Some are distance only without direction. Some are 1D (a line, usually horizontal) and some are 2D. Many objects are partially opaque, which is confusing. Resolution is very poor compared an optical device of the same size.
> Could multiple radar emitters and receivers be used to create a phased array and improve resolution?
Yes. However, this is currently bulky and expensive (in dollars and in compute power). Thankfully, it looks like capitalism is coming in to the rescue here and miniaturizing the everloving shit out of complex radar arrays for human interface tech. This should be usable for vehicles as well.
The Florida situation was the easiest set of circumstances for machine vision to handle and it still failed. More complex, dense road systems with real terrain are much harder to handle. This radar system will still get someone killed.
In the case they're talking about here, it's because to do so you need to predict where the non-visible road surface is going to be.
Consider travelling up a continous, slight incline. Precisely at the crest of the incline is an overhead gantry sign, positioned such that for an observer travelling up the incline it is located directly in line with their current direction of travel.
The only way that you "know" that the road actually dips under the sign rather than the sign being on the road surface is experience.
One of the only situations where both "brake" and "break" have the same meaning in a sentence.
Upon closer reading, he explains it somewhat.
https://twitter.com/elonmusk/status/771446048262946816
> Finishing Autopilot blog postponed to end of weekend
https://twitter.com/elonmusk/status/774155658476212224
> Will get back to Autopilot update blog tomorrow.
https://twitter.com/elonmusk/status/774664927835553792
> Will do some press Q&A on Autopilot post at 11am PDT tmrw and then publish at noon. Sorry about delay. Unusually difficult couple of weeks.
I wonder how many fatalities it'll take before they realise that 'the driver should've been paying attention' isn't a good enough excuse.
Note that humans use whitelisting too, to the point that they do not notice when a stop sign has been defaced or converted to a "give way".
> Curve speed adaptation now uses fleet-learned roadway curvature
So they have an advantage over Google and Apple in that they have mass market cars full of sensors being driven for them for free.
What your phone can send is only "I'm more or less at X, I'm moving fast so we may be in a car. But I may as well be in a glider. Who knows..."
(emphasis mine)
This has interesting privacy implications. I am not a Tesla owner, but I imagine that by enabling Autopilot you consent to providing Tesla with diagnostic, error, and sensor data. But what about those who have not enabled this feature? Their Tesla will automatically phone home with data regarding their location and surroundings regardless of whether or not they have consented to this?
As with any beta testing of software, high-fidelity diagnostic data is a must, especially in these circumstance as you have no other way of simulating the various scenarios people find themselves in when driving.
I'll trade my location privacy for awesome car capabilities.
In no way is it acceptable to give customers "experimental" software, overhype it, and disclaim responsibility when it inevitably fails.
But now with Tesla user-tracking, people seem to be actively psyched at being tracked by Tesla.
In this case, people mostly agree that R&D in self-driving cars is important, and can clearly see how this data helps with that. Whereas identifying traffic in Google Maps/Apple Maps feels less important, and the connection between location tracking and detecting traffic takes a little more work to understand.
Assuming of course that new Starbucks don't start showing up on routes preferred by Tesla drivers.
Privacy erodes naturally. It is inventible. The benefits outweigh the costs almost always. The need of the many outweighs the needs of the few.
The future is one in which humans are recognized, at least by the machines, as being more like a single organism than a group of individuals.
Data collection is all about machines being able to treat you as an individual.
There's someone making the same consent argument (https://news.ycombinator.com/item?id=6873947 ) and lots of people expressing ambivalence and more about sharing the data with Google.
As far as how people are responding to Tesla doing it, you did reply to a comment questioning their practices without evening being exposed to them.
The interesting thing is the data set of watching humans drive and using models to drive for the same place. This only works if the "place" is not notably different from the model, say a semi has hit the overhead and its now hanging into the roadway, can the car distinguish between a sign hanging sideways and one that is attached normally?
Severe storms and down power lines is another interesting question. Does autopilot recognize the environment has been grossly modified and refuse to drive? Earthquakes, tornadoes, floods, all can grossly change the environment at a particular geocoded location.
What if a Tesla owner's club decides to use a piece of highway 58 out in Nevada as a race strip? Does autopilot assume that when you hit this point you are supposed to stomp the accelerator and go as fast as you can? (ok that is a stretch)
It's the data without the knowledge. Something machine learning is bad at (hence turning chat bots into vitriol spewing fascists). VERY interesting times.
Humans are not perfect in those situations either, and cause plenty of fatal crashes.
Remember, autopilot doesn't have to be anywhere near perfect. It only has to be better than humans are now.
I used to think about it the way you do, but lately I think if they can make the numbers work out, then there will be some insurance companies willing to step in and turn risk of lawsuits into a manageable cost of doing business.
Or like x-ray machines?
Or like any other machinery that is automated and kills the odd person here and there?
These cases show us it doesn't need to be perfect at all.
As do other car accidents.
Perhaps there is a way to opt out of this - but also, if you opt out, do Tesla disable features of the car?
In the race to the self-driving car, Tesla now has one big advantage: they have tens of thousands of cars with the autopilot hardware driving around every day. This gives them a huge lead in the amount of data about their software performance - just comparing what the radar sees and how the human drives in any situation should make a difference.
Tesla will never come close to a stage 4 autonomous vehicle with the hardware rigs they're currently selling. That said, it'll be interesting to see what improvements they can make with software. Given their over-promise and under-deliver history though (which arguably killed someone), I'll take their marketing with a grain of salt.
Tesla is very explicit about the limitations of their current system. So the autopilot accident seems to be mostly about the owner not really understanding what the current autopilot can do and what not - after all, in the car it is just called "autosteer".
There is also already talk about the autopilot 2.0. This consists of augmented hardware, e.g. 3 different front-facing cameras. Only with that hardware Tesla is trying to reach level 3 or 4. The 8.0 update is about enhancing the quality of the existing autopilot but not about reaching new levels of automatic driving.
http://evanstonnow.com/story/public-safety/bill-smith/2016-0...
you could do the same with "normal" cameras (and probably lidar as well) i guess by pointing a laserpointer at it. the safest option is probably to just brake.
here elon musk says they don't use lidar: https://techcrunch.com/2016/09/11/tesla-autopilot-8-0-uses-r...
(as an aside, some years ago we had incidents in the news where people pointed laserpointers at aircraft pilots while landing. the pilot's appropriate response is usually to abort the landing and do a go-around, because that's the safest thing to do in this situation)
If data is the differentiating factor in this game, Google has less of it! Which is interesting position for Google to be at!
If you're talking about Uber and Left, I think Google currently has them beat in terms of data on a global scale, given how long they've been collecting data for Maps. You're talking about a future scenario where Uber and Lyft have rolled out significant numbers of sensor laden cars, but right now they don't have that - and who's to say Google won't have another approach by then.
Teslas definitely have many more miles on them, but I don't think the cars are sending back every single frame captured by its cameras back to Tesla HQ.
With this new usage of the radar, they seem to create "radar maps" of all radar echoes from bridges and traffic signs. Using those maps they should be able to detect true obstacles with a high enough confidence to enable automatic braking/evasion. The problem so far with automatic braking by radar is not so much with detecting possible obstacles but with false positives. A car must not randomly brake when there is no obstacle.
Im sure this will flip one day when if Google increases their map fleet
waze is how their maps product knows where the traffic is on surface streets.
Obviously you see Waze info on Waze itself but it has to work without any other users nearby. (Btw I don't think Waze and Google really share all their data, Waze doesn't even have lane routing.)
One thing that Tesla have is human input data. Google won't have that and will never have that I believe?
"Developers" ( in the sense of that Ballmer clip ) and sysadmins created a critical mass that carried Microsoft forward.
This is especially true of "Interface alerts are much more prominent, including flashing white border on instrument panel." This has been a huge thing in aviation automation for like ... forever.
Another question that popped in my head: what did the requirements look like that allowed such a huge (I assume) CPU and memory budget available that they could improve the system with "six times as many radar objects with the same hardware with a lot more information per object."
I wonder if they have heard of Six Sigma.
Six Sigma has been, in my view ( appeal to observer bias ) dominated by supply chain activities. There is too much delay variation in software for it to be much use.
(this being said, internalizing at least JIT to software seems pretty useful ).
It feels scary to discard millions of years of evolution and go with radar-first, but as always, time will tell.
Are there any visualizations available of what these radar patterns look like?
I wonder whether a human looking at the radar visualizations could reliably discriminate between them. I have to suspect that we could, but I've never seen such a visualization and don't have a good sense of what kind of detail it includes.
One of the most striking graphs I've seen was the plot of the transparency of water vs optical wavelength.
Basically, there's only a tiny transparency gap, which coincides almost exactly with the wavelengths the human eye is sensitive to.
http://hyperphysics.phy-astr.gsu.edu/hbase/chemical/watabs.h...
If you can bring your own well collimated coherent source you have no reason to stick to visible light. On the contrary, other spectra might be better (lower background noise, easier to produce coherently, easier to direct, etc).
Though possibly an argument could be made that a self driving car doesn't need to read signs, if it can get that data from a formatted online source. (Eg street names on signs, vs from GPS and maps)
To the downvoters: how do you expect to have an environment conducive to intellectually gratifying discussions, if you're not willing for someone to share a wild idea, and maybe even be a little bit wrong, without penalty?
I would guess that Musk or someone declared that cameras would be the primary approach and the engineers below kind of had to go along with the idea on the surface, but in the actual implementation some of that really came down to what the radar was saying anyway.
If this is ever going to be 'level 4' it will probably be with both an enhanced deep learning visual system (doesn't exist quite yet) and an as-yet-non-existent inexpensive elevated LIDAR. The LIDAR might come down in price within the next two or three years.
Wow, glad to see that they are using big data and machine learning. If all those 400k orders go through, there will be a network effect in favor of Tesla.
> With further data gathering, car will activate Autosteer to avoid collision when probability ~100%
> Curve speed adaptation now uses fleet-learned roadway curvature
So yeah, there's a precedent :)
If not... that would turn off a lot of privacy-conscious people, which Tesla doesn't tend to attract at its current prices but may become relevant as they come out with cheaper cars.
Sorry. Privacy kind of went byebye.
"You already have heart disease, who cares about colorectal cancer? Have a cheeseburger!"
Privacy isn't some binary choice, and having less data collected means less data being spilled when X company has a server compromised.
Except you can, you know, turn that off.
In any case, there's a difference between Google storing my location (they don't), Tesla storing my location (they would), and NSA storing my location (they do).
Google gives me the option to opt out. If I opt in, they give me cool features that require said location history.
NSA doesn't let anyone opt out, naturally.
Tesla doesn't let users opt out, even if they don't use autopilot. As we've seen, they actively use their location data to attempt to exonerate themselves whenever there's any kind of accident. In any case, Tesla gives me nothing in return for this information. If I were to try to modify the car to send no data, I don't doubt that Tesla would disable the fucking car.
It might not be anything like Tesla autopilot, but it's still a pretty sweet taste of the future. So stands to reason more can be done with it; I wish I had the funds for a Tesla... Maybe one year :)
What I'd like to see is an emphasis on stupefyingly comprehensive test vectors.
Seems this is a well-known problem in airborne radar: http://www.dtic.mil/dtic/tr/fulltext/u2/a402557.pdf
combined with
> ...we now believe [radar] can be used as a primary control sensor without requiring the camera to confirm visual image recognition.
seems like they're now intentionally ignoring the possibility of wooden and painted plastic obstacles?
I guess I'm surprised that what sounds like a large change in ConOps can be rolled out as an upgrade across a fleet in such a short period of time. It'd be fascinating to hear what sort of V&V had to be done, and how it was accomplished so quickly, to make this happen.
I mean, a really cynical view looks at the part where it works better as a smokescreen for the part where it much more aggressively monitors driver attention.
Unless he means "UFO" in the formal, aviation sense of an unidentified radar contact, I think Mr. Musk is severely underestimating alien stealth technology. I can't quite wrap my head around the notion that an advanced species could perfect interstellar travel but somehow be incapable of stealth technology.