Video of Tesla FSD almost hitting pedestrian receives DMCA takedown
twitter.com
twitter.com
The dotted black line coming from the front of the car (which I am assuming is the intended route) quickly snaps from straight ahead, to a hair pin right, to a normal right turn.
Ignoring the fact that the right turn happened to be into a pedestrian crossing with people on it - what was the car even trying to do? The sat-nav shows it should have just continued forwards.
I am astounded that software capable of these outputs is allowed on the roads. When could crossing a junction then taking a hairpin right back across it, ever be the correct thing to do?
On the other hand, it's always terrified me how jittery any version of AutoPilot's perception of the world is. Would you let your car be driven by someone with zero object permanence, 10/20 vision and only a vague idea of the existence or properties of any object other than lane markings and number plates?
Tesla's over the top marketing and hype seems at once to have been a key ingredient in their success, but also so frustrating because their product is genuinely awesome. I've long been kind of a skeptic but I could not have been more impressed with the test drive I took. It had me ready to buy in about 90 seconds. I wish there were actually-competitive competitors with similar range and power that aren't weird looking SUVs, from brands that don't lean into pure hype.
> Presumably you are aware that the visualization you see in any retail car is old software, and several iterations behind what the linked video is about (FSD Beta v10). Plus the visualization is quite a bit different (and in many ways simplified) versus what's used for the actual piloting in retail vehicles.
How do you know that what is on the screen matches what the system is 'thinking'? What reason or obligation would Tesla have to engineer accurate visualizations for the real thing and show them to you (remember the definition of accuracy: correct, complete, consistent)? Would Tesla show its customers something that would make them uncomfortable or show Tesla in a bad light, or even question the excitement Tesla hopes to generate?
I think it's likely that the display is marketing, not engineering.
I guess it thought that the road ahead is too narrow to continue. But if this is the case why did it simply not stop?
I think this is a navigation issue. This is exactly what I would have done if I had a passenger yell "WAIT TURN RIGHT TURN RIGHT oh nevermind GO STRAIGHT"
I’m stunned that it hasn’t, so far as we know, actually hit anything yet. I’m not sure how, but clearly they’re doing something right between choosing drivers and writing software.
Teslas driving themselves hit objects all the time.
1. https://www.google.com/amp/s/abcnews.go.com/amp/Business/tes... 2. Ibid 3. https://www.google.com/amp/s/www.mercurynews.com/2020/07/14/... 4. https://www.google.com/amp/s/amp.cnn.com/cnn/2021/08/30/busi...
But, I believe this discussion is about the new FSD software which is supposed to be more capable. Have we had reports about the new one doing the old tricks?
Right, a strawman.
How about this compromise: Let's call it "Poorly Implemented Self Driving" until it improves, and we won't try to sell it years early too.
Surely.
The car was driving 11 mph. The obvious thing to do is very simple: hit the brakes! Stop, re-evaluate, and then continue.
What exactly would the machine evaluate in a couple seconds that it can’t do instantly?
Sure - which is why I said I agreed there should be some indication of it being in FSD or not, originally.
I do not believe that Tesla's technology will ever achieve "Full self driving" (levels 4 or 5, where a driver isn't required https://www.nhtsa.gov/technology-innovation/automated-vehicl...), and the labeling of their "conditional automation" system as "full self driving" is an absolute travesty.
People do honestly think they have bought a self driving car. The marketing lie absolutely tricks people.
Any of those could mean a car could randomly turn into the crosswalk.
Humans are horrible at assessing risk. We worry about Teslas because it's on the headlines now, but you're infinitely more likely to get ran over by one of the cited examples than a Tesla just by sheer statistics.
TBH I don't think it's ever crossed my mind whether a Tesla was in FSD or not, even when driving alongside one. As long as it doesn't enter my space, I'm fine.
Walking through Manhattan, I'm more worried about the human driver nowadays than the non-human.
People say "how could it happen?!" and get angry for a software bug - that can be fixed - and seem to accept how drunk drivers do this (and a lot worse) literally every single day. But since it's a human, that's fine!
Edit: wording
What I mean is that you don't see a headline on news outlets or twitter hashtags trending for every drunk driver fatality or crash that happens. Which for me tells that collectively, we are fine with it. We accept it as ordinary, as crazy as it may be.
My comparison with drunken drivers is just with the regards to the odd behavior. If you look at a drunk driver (or even someone that fell asleep in the wheel) with a near crash, many times it would resemble this video. But the outrage from the public differs vastly.
Unpredictable death-robots roaming the streets are pretty novel.
I don't think that's very complicated, from a news perspective.
I've certainly seen many more local "person dies in car accident" non-brand-related stories than "person dies in Tesla accident" stories.
That doesn't mean we need to endorse putting more cars on the road that sporadically act like drunks.
You could make a law that states that your FSD has to be at least as good as humans. That means you have the same post-problem verification but now the parallel with drunk drivers can be made.
On the other hand, it's not like the software is always bad and always in the same situation. That is a big difference with a human analogy; a drunk driver taking a trip in the car is drunk for the entire trip (unless it's a very long trip etc..), so would be impaired for the entire duration.
There are plenty of people and vehicles that are road hazards and are allowed to drive (or be driven) anyway, so if we really cared about that aspect on its own we could probably do with better tests and rules in general.
The fact that enforcement is patchy is irrelevant — drunk driving is deemed serious enough to be an automatic infraction.
Also, most of the time drunk drivers are not actually that bad at moment-to-moment driving. That’s why almost everyone worldwide used to do it! You can still do the basics even when reasonably drunk. That doesn’t make you safe to drive. It’s still incredibly dangerous to do.
I think that some form of responsibility has to be assigned to the FSD in some way (the manufacturer? the user? some other entity or a combination?) regardless but I haven't found any clear case of how that would work.
It also makes me wonder how we would measure or verify a human driver with intermittent drunkenness. Imagine 5 seconds out of every minute you temporarily turn into a drunk driver. That's plenty of time to cause a major accident and kill people, but on the other hand that would mean that the combination of a situation where that would happen and the right timing to not be able to judge that situation has to apply. We do of course have the luxury of not having humans constantly swapping drunk and normal driving, so it isn't a realistic scenario, but it would make for a better computer analogy.
Besides drunk drivers we also have just generally crappy drivers that just happened to get lucky when doing their driving test (although there are places where no meaningful test is required so that's a problem in itself).
- drunk drivers are also not uniformly awful drivers: they can drive OK for the most part
- they still drive unacceptably poorly
- we strictly punish them on detection, before any potential accident
- drunk drivers and FSD have more in common with each other than competent drivers and FSD
- why is FSD not held to such a preventative standard?
One can argue that FSD is like a drunk driver driving a student training car with two sets of pedals and two steering wheels, and the Tesla owner/driver is like a driving instructor. But driving instructors are trained and paid to be quite vigilant at all times. Tesla play a sleight of hand and say it’s a labor saving technology, but also you need to be able to behave like a trained and paid driving instructor... that is a conspicuous contradiction.
And I’m ignoring future FSD capabilities because while I’d be happy for it to come about, we should discuss the present situation first, and I don’t believe it’s a good example where sacrificing lives now is acceptable in order to potentially save lives in the future.
The same can't apply to FSD because it isn't consistently 'driving drunk'. That analogy doesn't hold because it is not fixed software like a GPS-based navigation aid would be. Just like humans it does have more or less fixed parameters like the amount of arms and legs you have, that doesn't tend to change depending on your intoxication.
One could make the argument that it's not as much the "haha it is just like a drunk driver zig-zagging", but the uncertainty about the reliability. If a car with some autonomous driving aid drives across an intersections just fine 99 times out of a 100, and that one time it doesn't, that doesn't mean the car software was dunk 100% of the time.
Why FSD is not held to some standard, I don't know. I suppose that depends on how it is defined by local law and how the country it is in allows or disallows its use.
The problem with prevention and detection here is that like humans, the system is not in a static state. The trained neural network might be largely the same with every release, but the context in which it operates isn't, unless the world around it stops completely in which case two trips can be identical and because the input is identical the output can also be identical. Humans do the same, even if well-rested and completely attentive, knee-jerk reactions happen.
Holding FSD to a standard of a drunk driver isn't a valid comparison due to the non-static nature of the state it is in. This isn't even FSD-specific, even lane guidance/keeping assistance and adaptive cruise control isn't static, and those are based on pretty static algorithms. Even the PID-loops used on those will deliver different results on seemingly similar scenarios.
Perhaps we should stop comparing technology to humans since they are simply not the same. The static kind isn't and neither is a NN-based one. We can still explore results or outcomes because those are the ones that have real impact. And let's not fool ourselves, humans are far less reliable in pretty much every man-machine combination. But in human-to-human contexts we factor in those unreliabilities, and with machine-to-human or machine-to-machine we seemingly don't, which is pretty much the same problem you're describing.
This will be an interesting field of development, and if we simply take death toll into account, keep in mind that for some reason seatbelts were thought to have 'two sides of the story' as well when they were first introduced and later required. As with car seats for children and infants, a good idea might start out one way and over time (with the accompanying bodycount) it gets shaped into whatever we expect of it today. Same goes for aerospace, boats and trains, and that's even without taking software into account.
Certain medical conditions are analogous to this. This is handled in various ways, including doctors reporting you to the state DMV and suspending your license if your condition is not very well controlled.
I'm absolutely certain that they've attempted to simulate millions of hours of driving, yet this odd behavior happened in real life, which now can be studied and fixed.
If we never had the software on the wild, how would you ever really test it?
Genuine response:
1 don't let customers do the testing, especially if you don't train them (I mean real training about failures not PR videos and tweets and some small letter manual with disclaimers)
2 use employees, train drivers to test, have some cameras to check the driver to make sure he pays attention.
3 postpone testing until the hardware and software is good enough so you don't ignore static objects.
4 make sure you don't do monthly updates that invalidates all your previous tests.
Unfortunately it's always about the incentives and on a capitalist society the only incentive is money. So even if they could be safer, they wouldn't do it unless it's more profitable, specially being a publicly traded company.
After all, normal (complex) software tends to have lots of in depth details you need to test for; and a surface area that's pretty irregular in the sense that it's hard to do generalized testing. Some bits can be fuzz tested, but usually that's pretty hard. It's also quite hard for a generalized test to recognize failure, which is why generalized test systems need lots of clever stuff like property testing and approval testing, and even then you're likely having low coverage.
However, a self-driving car is amendable to testing in a sim. And the sim might be end-to-end, but it needn't be the only sim you use; the FSD system almost certainly has many separate components, and some of those might be easy to sim for too; e.g. if you have a perception layer you could sim just that; if you have a prediction system you might sim just that; etc.
And those sims needed be full-sim runs either; if you have actual data feeds, you might even be able to take existing runs, and the extend them with sims; just to test various scenarios while remaining fairly close to real world.
I'm sure there are tons of complexities involved; I don't mean to imply it's easy - but it's probably tractable enough that given the overall challenge, it's worth creating an absolutely excellent sim - and that's the kind of challenge we actually have tons of software experience for.
The onus is on the company trying to do this to figure out a safe way.
They don't (should not) get to test in production with real innocent lives on the line just because they can't come up with a better answer.
The problem is too hard for existing tools. Good job on the 80% case; let's look for problems for which the 80% case is good enough and back away from unsupportable claims about "just a little more time" bringing FSD in environments defined by anomaly and noise, in which the cost of error is measured in human life and property damage.
With self driving it takes longer for the one individual to learn, but the difference is that there is only the one individual, and it need not ever die. The learning is transferable between cars, regressions between generations need not be accepted. The problem of bad driving can finally be solved.
Id say that yeah, it seems that the car detected these signs and read them as "you can only go that way" even when the street was open
As to abrupt maneuver, EU limits rate of change for steering control and I think it would be wise for US to adopt something similar for driver assist systems.
That's a terrible idea. First it's a bandaid over an underlying problem. Second it's a legislative answer to a technical problem, which is IMHO never a good idea.
Above everything, the car's driver assist should not be making moves that human does not understand if system requires human to supervise it.
Let’s not go down the rabbit hole of government specifying how machines we haven’t even built yet should work.
If evasive maneuver is required it would have to be done in fully autonomous way. For the time of the maneuver system would have to be responsible for driving.
Me too. However, in my encounter with software like this, it usually runs on meat brains.
I don't have answers, but it does seem to me like we are not placing a premium enough on structuring this tech to optimize driver supervision over the driving behavior. Granted, the whole point is to one day NOT HAVE to supervise it all, but at this rate we're going to kill a lot of people until we get there.
If a GC has to pause the world to do its job there's none of that background processing happening while it's "looking away".
What if say, a system is written against an indeterminate timed GC like say, Azul or Go's, but code is written in a way that proves GC times never exceed X, whether by theory or stress testing. Is this still seen as 'unguaranteed'?
Then you need: not-blocking data strcutures, not just lock-free - that are much easier to develop. Pretty much you need forward guarantees on any data structure you'd use.
So I kind of would agree that producing any garbage during runtime makes it much harder to prove that a program can fulfill hard realtime guarantees.
there are keys difference between automation in e.g. aircraft, and what Tesla at al are failing at,
e.g., how constrained the environment is; and what the exposure is to anomalous conditions is; and what the opportunity window usually is to turn control back over to a human.
The thing I don't understand is, we have a much more comparable environment in ground travel: federal highways.
Innumerable regressions and bugs and lapses aside, I do not understand why so much effort is being wasted on a problem which IMO obviously requires AGI to reach a threshold of safety we are collectively liable to consider reasonable; when we could be putting the same effort into the (also IMO) much more valuable and impactful case of optimizing automated traffic flow in highway travel.
Not only is the problem domain much more constrained, there is a single regulatory body, which could e.g. support and mandate coordination and federated and data sharing/emergent networking, to promote collective behavior to optimize flow in ways that humans limited-information self-interested human drivers cannot.
The benefits are legion.
And,
I would pay 10x as much to be able to cede control at the start of a 3-hour trip to LA, than to be able to get to work each morning. Though for a lot of Americans, that also is highway travel.
Not just this, why not start with the low-hanging case of highway travel, and work out from there onto low-density high-speed multi-lane well-maintained roads? Yes that means Tesla techs who live in Dublin don't get it first. Oh well...
IMO there will never be true, safe FSD in areas like my city (SF) absent something more more comparable to AGI. The problem is literally too hard and the last-20% is not amenable to brute forcing with semantically vacuous ML.
I just don't get it.
Unless we take Occam's razor, and assume it's just grift and snake oil to drive valuation and investment.
Maybe the quiet part and reason for engineering musical chairs is just what you'd think, everyone knows this is not happening; but shhhh the VC.
Anywhere else is basically a parlor trick. Yes, it sorta works, a lot of the time, but you have to monitor it so closely that it isn't really beneficial. As you point out, its going to take some serious advances (which in all likelihood are 30+ years away) for FSD to reliably work in city centers.
I think the issue you've highlighted is one of governance. There's only so much Tesla can do regarding highways. You really need the government to step in to mandate coordination of the type I think you're envisioning. And the government is pretty much guaranteed to fuck it up and adopt some dumb standard that kills all innovation after about 6 months, so it never actually becomes usable.
I think automakers will eventually figure this out themselves. As you say, there are too many benefits for this not to happen organically. Once vehicles can talk to each other, everything will change.
And people paid money for this privilege?
And I'm now trying to remember, but I think autopilot was initially free (or at least was included with every car I looked at, so it didn't seem like an extra fee). Auotpilot is now standard on all Teslas, but FSD is an extra $10k, which IMO is a joke.
I'm not paying with money and my life to be a corporation's guinea pig.
Also, the problem is “easy” for the general case, but the edge cases are almost singularity-requiring. The former is robot vacuum level, the latter is out of our reach for now.
is why there we haven't seen the feds stepping in via the transportation agency to develop and regulate exactly this, with appropriate attention paid to commercial, personal, and emergency vehicle travel.
The opportunities there appear boundless and the mechanisms for stimulating development equally so...
I really don't get it. Then I think about DiFi and I kind of do.
I think there's some of that. Some overconfidence because of the advances that have been made. General techno-optimism. And certainly a degree of their jobs depending on a belief.
I know there is a crowd of mostly young urbanites who don't want to own a car and want to be driven around. But I agree. Driving to the grocery store is not a big deal for me. Driving for hours on a highway is a pain. I would totally shell out $10K or more for a full self-driving system even if it only worked on interstates in decent weather.
as you say, i don’t think this is entirely out of reach (even if it required specialized highway infrastructure or car to car communication). seems like lower hanging fruit than trying to get full self driving working on local/city roads.
i would pay a ton for the highway only capability…
Driver assisted highway has been around for years... Level 5 driving requires no human attention. Huge difference.
I think what is wanted by many is level 5 on the highways. I want to sleep, watch a movie, whatever. That is much, much "better" than what we have now. Like many others, I would be most interested in full level 5 on highways and me driving in the city. That is also much easier to implement and test. The scope of the problem is greatly reduced. I think Tesla and others are wasting tremendous resources trying to get the in-city stuff working. It makes cool videos, but being able to do other activities during a 5 hour highway drive has much more value (to me at least) than riding inside a high risk video game on the way to work.
(edit) I get that I am misusing the definition of "level 5" a bit, but I think my meaning is clear. Rated for no human attention for the long highway portion of a trip.
I would appreciate a simple "keep-distance-wrt-speed" function for bumper-to-bumper situations. Where worst case scenario, you rear-end a car at relatively low speeds.
I'd happily keep control over steering in this situation and just keep my foot over the brake, though lane-keep assist would probably be handy here as well. A couple radar sensors/cameras/or lidar sensors would probably be enough for basic functionality.
Disable if the steering-wheel is turned more than X degrees - maybe 20 or 25?. Disable if speed goes over X speed - maybe 15mph? Most cruise controls require a minimum speed (like 25mph) to activate.
Trying to do full driving automation, especially in a city like Seattle, is like diving into the ocean to learn how to swim.
As cool as that sounds, I'd trust incremental automation advancements much more.
This has been widespread for at least a decade. I'm not even aware of a mainstream auto brand that doesn't offer adaptive cruise control at this point. Every one I've used works in stop and go traffic.
The other features you want are exactly what Tesla has basically perfected in their autopilot system, and work almost exactly as you describe (not the FSD, just the standard autopilot).
I can't say that my experience agrees with this. Maybe some higher-end vehicles had it a decade ago, but it seems to be getting more popular over the past 5 years or so. I still don't see it offered on lower priced vehicles where basic cruise functionality is there, but I doubt ever will be a part of "entry level" considering the tech required.
None of the vehicles my family owns have a "stop-and-go" cruise control function - all newer than 10 years. ACC at higher speeds is available on one, but it will not auto-resume if the vehicle stops.
I think an even more 'sensor fusion' approach needs to be adopted. I think the roads need to be 'smart' or at least 'vocal'. Markers of some kind placed in the roadway to hint cars about things like speed limit, lane edge, etc. Anything that would be put on a sign that matters should be broadcast by the road locally.
Combine that with cameras/lidar/whatever for distance keeping and transient obstacle detection. Then network all the cars cooperatively to minimize further the impacts of things like traffic jams or accident re-routing. Perfect zipper merges around debris in the roadway.
Once a road is fully outfitted with the marker system, then and only then would I be comfortable with a 'full autopilot' style system. Start with the freeway/highway system, get the logistics industry on board with special lanes dedicated to them and it becomes essentially a train where any given car can just detach itself to make it's drop off.
A more advanced problem would be safely stopping in case the driver falls asleep/looses consciousness, eg. on a highway. that short amount of self-driving is less error-prone than the alternative.
Cars have a vastly harder job because they're navigating through an environment that is orders of magnitude more complex because there are other moving objects to deal with.
Unexpected things can happen on highways but a lot fewer of them and it's not like humans driving 70 mph are great at avoiding that unexpected deer or erratic driver either.
ADDED: You'd actually think the manufacturers would prefer this from a liability perspective as well. In a busy city, pedestrians and cyclists do crazy stuff all the time (as do drivers) and it's a near certainty that FSD vehicles will get into accidents and kill people that aren't really their fault. That sort of thing is less common on a highway.
We could just expand MARTA, but the NIMBY crowd won't allow it. People are still hopped up on 1980s fearmongering about the sorts of people who live in cities and don't want them infesting their nice, quiet suburbs that still have the Sherriff posting about huge drug and gun busts.
For me, it's more the 3-5 hour drive, mostly on highways, to get up to northern New England.
This is why there are SAE levels of automation and specifically Level 4 is what you're describing. Anyone claiming their system will be Level 5 is flat out lying.
Is that actually the case, though?
I would hope, although perhaps I'm mistaken, that the developers of the actual self-driving systems would be able to effectively reason about what's happening. For example, would a senior dev on Tesla's FSD team look at the video from the article and have an immediate intuitive guess for why the car did what it did? Or better yet, know of an existing issue that triggered the wacky behavior?
Even if not, I'd hope that vehicle logs and metrics would be enough to shed light on the issue.
I don't think I've ever seen a true expert, with access to the full suite of analytic tools and log data, publish a full post-mortem of an issue like this. I'm certain these happen internally at companies, but given how competitive and hyper-secretive the industry is, the public at large never sees them.
Aircraft operate in an extremely controlled environment that is almost embarrassingly simple from an automation perspective. Almost everything is a straight line and the algorithms are intro control theory stuff. Lateral nav gets no more complicated than the difference between a great circle and a rhumb line.
The collision avoidance systems are cooperative and punt altogether on anything that isn't the ground or another transponder-equipped airplane. The software amounts to little more than "extract reported altitude from transponder reply, if abs(other altitude - my altitude) < threshold, warn pilot and/or set vertical speed." It's a very long way from a machine learning system that has to identify literally any object in a scene filled with potentially thousands of targets. There's very little to worry about running into in the sky, and minimum safe altitudes are already mapped out for pretty much the entire world.
Any remaining risk is managed by centralized control and certification, which just isn't going to happen for cars. We aren't going to live in a world where every street has to be (the equivalent of) an FAA certified airport with controls to remove any uncertainty about what the vehicle will encounter when it gets there. Nor are we going to create a centralized traffic control system that provides guarantees you won't collide with other vehicles on a predetermined route.
So it's just a completely different world with completely different requirements. Are there things the aerospace world could teach other fields? Yeah, absolutely. Aerospace is pretty darn good at quality control. But the applications themselves are worlds apart.
I'm with you that they are super different, and that the auto case is likely much, much harder. But I see that and can't help but think that the path we should be following here is one with a much greater and healthy skepticism (and far greater human agency) in this automation journey than we are currently thinking is needed.
Airplanes are responsible for 200-300+ lives at a time, so it’s quite incomparable to road vehicles. Of course it makes sense to have human oversight in case something goes wrong.
On the flip side, the average car driver is not very skilled nor equipped to deal with most surprises - hence the ever present danger of road traffic.
I’m not sure why AI drivers are held to such insanely high standards.
See: https://en.wikipedia.org/wiki/Motor_vehicle_fatality_rate_in...
Sorry, most drivers I’ve seen avoid the kid that follows the bouncing ball into the street and all the other random events. A few drivers are crap. But justifying automation based on that few is lazy and incorrect. Elon is just full of his own crap on Tesla’s autopilot.
I've seen many poor drivers over the decades, but can't think of any human drivers who would reliably and repeatably crash into a stationary fire truck in their lane. "Full Self Driving" vehicles, on the other hand...
https://www.bloomberg.com/news/articles/2019-09-06/spotting-...
https://www.wired.com/story/tesla-autopilot-why-crash-radar/
https://slate.com/technology/2021/08/teslas-allegedly-hittin...
actually ive been thinking this is exactly the win self driving vehicles have been looking for. upgrade, certify, and maintain cross country interstates like I-70 for fully autonomous, no driver vehicles like freight, mail, hell even passengers. maybe that means one lane with high-vis paint and predefined gas station stops and/or some other requirements. i bet the government could even subsidize with an infrastructure spending bill, politics notwithstanding.
there cant possibly be a problem with _predefined_ highways that is harder to solve than neighborhood and city driving with unknown configurations and changing obstacles. i feel like everyones so rabid for fully autonomous Uber that the easier wins and use cases are being overlooked.
The problem with this is - why not just use trains at this point? Trains already an economical solution for point to point transportation.
Yup, that's a train. I really do wish US rail hadn't been turned into what amounts to private property. I live along the Amtrak Cardinal line and would love to use it to travel. But the low speed and frequent stops for higher-priority freight mean a trip takes longer than driving and usually costs more than I would pay in gas.
I do sometimes wish we'd just devote one lane of our larger freeways to self-driving cars exclusively. You could let the cars drive as fast as they want, and charge per-mile tolls for any car that uses it when the road is congested. Ideally, you'd charge based on occupancy as well: an empty car should have to pay more than a car with a human in it, and a car with 2+ people might be allowed to use the lane for free.
What aircraft autopilot does is following a pre-planned route to the T, with any changes being input by humans. The aircraft autopilot doesn't do its own detection of obstacles, nor of router markings; it follows the flight plan and reacts to conditions of the aircraft. Even when executing automatic take-off and landing, the autopilot doesn't try to detect other vehicles or obstacles - just executes the plan, safe in knowledge that there are humans actively monitoring for safety. There is always at least two humans in the loop: the pilot in command who prepared and inputed the original flight plan and also inputs any route changes when needed (collision and weather avoidance), and an air traffic controller that continuously observes flight paths of several aircrafts and is responsible for ensuring safe separation between aircraft in his zone of responsibility. Beyond that, an ATController has equal influence on all aricraft in his zone of responsibility, and in case one does something unexpected, it can equally well redirect that one or any other one in vicinity. Lastly, due to much less dense traffic, the separation between aircraft is significantly larger than between cars [1] providing time for pilots to perform evasive maneuvers - and that's in 3d space, where there are effectively two axes to evade along.
Conversely with car FSD - the system is tasked both with following the route, and also with continuously updating the route according to markings, traffic, obstacles, and any contingencies encountered. This is a significant difference in quantity from the above - the law and the technology demands one human in the loop, and that human can only really influence his own car at most. Even worse, due to density of traffic, the separation between cars is quite often on the order of seconds of travel time, making hand-over to driver a much more rapid process.
I am moderately hopeful for FSD "getting there" eventually, but at the same time I'm wary of narrative making unwarranted parallels between FSD and aircraft autopilot.
[1] https://www.airservicesaustralia.com/about-us/our-services/h...
The US govt isn't going to do anything until there's a public outcry, and historically there won't be a public outcry until there's a bunch of famous victims to point to.
I think this attitude is defeatist and absolves us of doing anything. It's a democracy; things happen because citizens act. 'The US government isn't going to do anything' as long as citizens keep saying that to each other.
I think it’s unfair to lump all self driving car manufacturers together.
The traditional car companies have been doing research for decades (see for example https://en.wikipedia.org/wiki/VaMP), but only slowly brought self-driving features to the market with part of the slowdown because they are aware of the human factors involved. That’s why there’s decades of research on ways to keep drivers paying attention and/or detecting that they don’t.
“Move fast and break things” isn’t their way of working.
The sat nav didn't update the route to go that way. The FSD system decided to. It probably completely lost track of the road ahead because of the unmarked section of the road and just locked on to the nearest road it could find, which was that right turn.
I've seen videos where it cross over unmarked road and then wants to go into the wrong lane on the other side because it saw it first. It seems like it will just panic search for a road to lock onto because the greatest contributor to their navigation algorithm is just the white lines it can follow.
There is a bigger lesson in there: click-driven video selection creates a very warped view of the world. The video stream is dominated by 'interesting' events, ignoring the boringly mundane that overwhelmingly dominates real life. A few recent panics come to mind.
When I drive, I don't spend 1 minute plowing into pedestrians for every 1 hour and 39 minutes I spend driving normally.
If a FSD Tesla spends just 99% of its time doing boring, non-click-worthy things, that is itself interesting in how dangerous it is.
To your point, I'm definitely interested in knowing how many minutes of boring driving tend to elapse between these events. The quantity of these sorts of videos that have been publicized recently gives me the impression that one of these cars would not be able to spend 100 minutes of intervention-free driving around a complex urban environment with pedestrians and construction without probable damage to either a human or an inanimate object.
FSD should not be allowed on the road, or if it is it should be labeled as what it really is: lane assist.
What I am saying is that selection bias has exploded in the age of viral videos, and this phenomenon doesn't receive anywhere near the attention it deserves. We can't make sound judgements based on online videos, we need quality data.
That's up to you (or Tesla) to prove, isn't it? Taking your (or Tesla's) word that it's good most of the time is utterly meaningless.
For example, short, wavy hills in the road would often crest just high enough that once it couldn't see on to the other side, it would immediately start veering into the oncoming lane. I have no idea why it happened, but it did, and it still wasn't fixed when I turned in my car in 2019. I drove on these roads constantly so I learned to just turn off AP around them, and it helped traffic on the other side was sparse, but if those weren't true, I'd only have a literal second to response.
EDIT: IMO the best thing it could do in that scenario is just continue on whatever track it was on for some length of time before its focus was lost. Because it "sees" these lines it's following go off to the right/left (such as when you crest a hill, visually the lines curve, or when the lines disappear into a R/L turn) but only in the split second before they disappear. Maybe that idea doesn't fit into their model but that was always my thought about it.
It's actually fascinating to watch - it just clearly has no semantics of "I'm in a lane, and the lane will keep going forward, and if the paint fades for a bit or I'm goign thrur a big intersection, the lane and the line of the lane is still there."
It also doesn't seem to have any very long view of the road. I got a Tesla at the same time of training my boys to drive, and with them I'm alwys emphasizing when far away things happen that indicate you have increased uncertainty about what's going on down the road. (Why did that car 3 cars ahead break? Is that car edging towards the road going to cut infront? Is there a slowdown past the hill?) The Tesla has quick breaking reflexes but no sign of explicitly reasoning about semantic layer uncertainty
The actual routing comes from something like Google Maps that is constantly evaluating driving conditions and finding the optimal route to the destination. It does this based on the vehicle's precise location but irrespective of the vehicle's velocity or time to next turn.
The actual AI driving the car is trying to find a path along a route that can change suddenly and without consideration. It's like when your passenger is navigating and says "Turn here now!" instead of giving you advanced notice.
If such a trivial and obvious edge case as navigation changing during the route isn't handled, it just shows how hopelessly behind Tesla is.
Well, great news for you then! Elon Musk just announced they're adding a button to allow a lot more people to get access to this beta software. https://twitter.com/elonmusk/status/1438751064765906945
Form the videos I have seen this fails often (in that the path doesn’t go straight even if it can). Knowing a bit about randomized met heuristics myself, I am not surprised.
I think, they need to perform some post processing on these paths (a low iteration local search).
I think, they should also start with a guess (like just go straight here, or do a standard turn), and then check if the guess is ok. I think, that could help with a lot of the problems they have.
Situations like these are sent to Tesla to be analyzed, the corresponding data is added to the training data, and the error is corrected in the next versions. This is how the system improves.
After this situation is fixed, there will be an another edge case that the HN crowd panics over.
This is incorrect. Tesla also relies on maps for traffic signs, intersections, stop signs etc. They just don't have additional details like the others do.
> After this situation is fixed, there will be an another edge case that the HN crowd panics over.
Is anything Tesla FSD can't handle an "edge case" now? It's literally an everyday driving scenario.
It isn't possible to learn edge cases without a lot of training data, so I don't see any other way.
I'm not saying that safety couldn't be improved, for example by disengaging more easily in situations where it isn't confident. One heuristic would be when it changes route suddenly, like in this scenario.
Also, the NHTSA dataset will contain old as hell cars, comparing it to a fresh out of the factory one will by itself skew the data.
It’s something I’ve learned to deal with but the sudden attempt to break from 70 to 55 is pretty bad especially as it’s unexpected for other drivers around you.
While I’m sure Audi are much worse at updating their software to fix known issues than Tesla I find myself skeptical that the mix of hacks cars use to implement these features scale well, especially in construction zones. Hence I’m pretty content with radar based cruise control and some basic lane maintenance and then doing the rest of the driving myself.
I can imagine if my car were slightly better at some of these things I’d be a significantly worse safety driver as I’d start to be lulled into a false sense of security.
That might explain the car trying to re-route on the display, although I'm not even sure it does. To me it looks like the black line disappears into compression artefacts at just the right time.
Mostly I think we don't have enough evidence either way and speculating just expresses our desire for the technology to work or not.
Crazy how it handled a shitty situation like around 6:55 in https://streamable.com/grihhc but failed there. Guess that shows what most of the ADAS engineers already know, the really hard part are those last few percentage points to make it actually reliable.
Is the reason why applying Silicon Valley hubris to self driving type problems is not going to work.
It's not an exaggeration to say, in 2050 ...We still be beta testing FSD on Mars roads :-)
With safety systems you always want multiple levels of overlapping checks.
We know how to make computers navigate maps perfectly. We don't know how to make computers see perfectly. The other uncontrollable humans on the map just make it worse.
Yes this'll not work in existing cities. That's fine. That's the point. An inscrutable black box "ML model" that can be fooled by things that humans aren't fooled by should not be in a situation where it can harm humans. I as a pedestrian did not consent to being in danger of being run over by an algorithm that nobody understands and can only tweak. Build new cities that are self-driving first or even self-driving only, where signaling is set up as I wrote in the first paragraph so that a car reliably knows where to drive and how to not drive over people. Take the opportunity to fix the other problems that old cities have like thin roads and not enough walking paths.
There was a video by one Tesla user over the past week that talked about how their Tesla v10 FSD software would attempt to turn into driveways, repeatedly, when driving down a straight road.
The user did a recalibration of their cameras, rebooted the car, and the problem went away.
Edit: to be clear, I plan to buy an EV in the future, but I'll drive it myself. Most of my driving is in a 15Km radius, urban, and babysitting the self-drive seems like more trouble than it's worth.
This is on top of Tesla classifying FSD as level 2 system (while promoting as “level 5 soon”) so they don’t have to share data with CA DMV. Only reason not to be transparent like the others is if you’re not confident in your system.
I think the biggest problem I have is that the driver doesn't intervene/even really stop the car. The good autopilot 'test channels' like 'AI DRIVR' will routinely disengage if any safety concern is present.
The monorail test driver, in contrast, knows the monorail is a problem and was criticized for not correcting the car driving under the monorail, which is already illegal behavior.
https://twitter.com/elonmusk/status/1437322712339423237?s=21
From the nav you can see the car is trying to go straight but swerves around some invisible obstacle. I wouldn’t be surprised if this was a failure in their new “voxel based” collision avoidance.
That's the entire problem. These systems don't understand anything about the world. If you see a sign covered in snow you're going to recognize it as such and make your best guess as to what it says and weather you care. You probably don't need that information to drive safely because you have a lot of knowledge to fall back on. These things dont, so some folks want to improve GPS accuracy and maps and signage. That's not a viable solution on so many levels.
General AI is the only way forward for fully autonomous cars.
What about that instant before it figures out it's bad enough weather to shut itself off? What is the threshold?
On the other hand, there are a lot of videos of FSD Beta on youtube that show the car driving into the sun where it's almost impossible for a human to see, but the cameras detect all of it.
The first winter after I got my license, I was driving around at night on a road I hadn't been to before.. kind of in the middle of nowhere, unpaved unlit back roads.
I saw a sign, partially covered in snow, and had a hard time making sense of it. Why this odd shaped sign in the middle of nowhere? I really wanted to figure it out and fixated on it a bit too much.. only right after making it past the sign, I realized it's warning about a tight bend in the road right ahead. Almost made it into the ditch. Thanks, sign, I would have seen the curve just fine if I had just focused on the road ahead.
There are way too many edge cases in real world driving conditions because road traffic and signage changes from instant to instant.
As a driver I don’t know what’s round the next bend, could be road works or a vehicle collision.
There’s no technical reason a computer based driving system shouldn’t be more aware, it’s an implementation failure.
Every new EV from Rivian to GM to BMW uses the “Combined Charging System.” Tesla, chooses to continue to use their own proprietary design in the U.S. despite having CCS technology available. (and the implementation completed already - European Tesla models ship CCS - required by law.)
Tesla will sell you $10K beta software that wants to steer into pedestrians, that most choose to pay for because after delivery, the price doubles (so act fast!) Most who fork that kind of cash are not able to access all parts of this beta software, this software license cannot sold/transferred to the next owner regardless of what the maroney sticker says, but don’t worry, FSD is being launched next month.
If thats excusable, surely then everyone’s ok with vendor lock in for DC Fast charging unless one wants to shell out $450 for a slower “old school” Chademo adapter, the fact one cannot buy replacement parts for their car, and cannot repair their own vehicle.
Tesla - America’s Beta Test Success Story.
In particular, road maintenance is already expensive and often left to rot. Still, current driving is relatively robust - even if a few signs are damaged or go missing, no major issues will occur. Road signs keep working even during storms, even if there is no power, even if something has hacked national electronic systems, even if the GPS is not accessible.
Replacing all this with brittle, finicky, expensive, easy to attack en masse digital systems is not a great idea.
Exactly! No wonder those pesky horseless carriages never caught on!
So you see no qualitative difference between a couple of horse carriage crossing the North American plains, and an Interstate system (that requires no stopping) with gas stations, motels, parking lots, malls, suburbs?
Have you looked at maps of cities with roads and parking lots? Cities in north America are literally built / rebuilt around cars, cities in Europe heavily modified.
Or, conversely, that means you find any advance towards autonomy in cars, in their current form, such as lane assistants, or working well in good weather on interstates, etc. is completely worthless?
That is, if autonomous cars can improve safety or convenience on current infrastructure, they will prevail, and new infrastructure will develop with them in mind. But if they are only useful after such changes, they will likely linger for decades before any adoption.
Autonomy today is in the same situation that cars once were; even the current state of 'autonomy' has obvious advantages already and will only become more in the future.
Edit: How do you think cars were useful at their inception? Slow, with a range of a few miles, needed to be rebuilt every few miles, gas to be bought in pharmacies. Quite a bit had to change before cars became something practically useful beyond showing 'I am rich'.
No one is forcing manufacturers to do anything. They a perceive a business opportunity and take it.
Tesla (and fanboys) say that they are clear about the limited capabilities of Autopilot and FSD when in reality, they indirectly support all fan-videos by letting them live on YouTube while DMCAing critical videos.
They want you to believe that FSD is more capable than it truly is.
Examples?
Obviously, I don't know if that is true that, 1 the video was DMCAed and 2 that Tesla sent the DMCA takedown request.
There must be quite a lot of false alarms to filter out...
I no longer have my Tesla, but I stopped using AP before I sold the car, partly as a result of this behavior. And partly because phantom braking became pretty common for a number of overpasses we go under on our way to grandma's house, and it really scares the crap out of my wife & kids.
Aside from phantom braking, the best use case for AP, in my opinion, is to be a jerk and cruise in the middle lane of the freeway. It deals okay with that most of the time.
That's literally "trusting it".
As a human, you are attempting to learn the situations under which the car operates safely. You could be very conservative and say "it never works". The big problem is that the kinds of errors that occur in these systems are very different from the kinds of errors that humans expect. Even designers of autonomous systems can be surprised by emergent behavior. You say you can quickly learn the good and bad sides of it, however, that's not true. How much reliability do we expect from a system that, when it fails, leads to sever injury or death? ASIL-D is something like one failure in 10^9 hours [1]. That's 100,000 years.
You do not have that much experience with your Tesla, sorry. In principle, you might drive every waking hour for 10 years, never witness an accident, and you still not have an argument that passes even the most basic scrutiny for safety for critical systems. It could make a fatal mistake in the 11th year and that would not be a big surprise.
No one has that much experience with the ASIL-D components of any car. That's why I we must rely on regulation and standardization to help keep us safe. Safety isn't just "well I tried it a bunch under these conditions and didn't get hurt". And that's why what Tesla's strategy is reckless.
[1] https://en.wikipedia.org/wiki/Automotive_Safety_Integrity_Le...
Sometimes I wonder about the folks who think AP makes for a more relaxing drive. I think that's only true if the level of defensive driving it does matches well with the driver's innate driving style. I'm not prone to having close calls on the road because I avoid them well before they get exciting. I've ridden with people who just don't drive that way, though. They'd probably be okay with AP.
Additionally I've experienced cases of "phantom braking" (https://www.cbsnews.com/news/automatic-emergency-braking-in-...) which doesn't instill confidence in those features.
Is it possible you have older autopilot hardware that runs different/older software?
It wasn't even close. Look at where the ped is at 12 seconds, compared to the car. It's because he stopped the turn, and straightened the vehicle that the pedestrian stopped and looked up with concern (at 13 seconds). Even if the pedestrian broke out into a full run, they wouldn't have been in any real danger.
This sort of close interplay between pedestrians and cars is very common in cities.
The concern I have is that the "eye contact" element is lost. When in doubt of another motorists intentions, we make eye contact and look for acknowledgement or an indication of intent. This doesn't exist with FSD.
Edit: Some people have pointed out the car wasn't supposed to be turning right which I missed. If that's the case, the situation and my opinion is completely different.
It abruptly tried to do a right turn on a no-right intersection with no warning (map said it was supposed to go straight, it looked like it was going to do that right up to the last second).
Putting aside the "almost hitting a pedestrian", this is really dangerous behavior for an autonomous vehicle.
https://www.google.com/maps/@47.6175579,-122.3456341,3a,90y,...
Even Google Maps has a car turning right here. I've always read those signs that you can't turn right from the left lane. It's confusing, for sure.
https://mynorthwest.com/1186128/slalom-seattle-monorail-colu...
This article seems to agree (and bonus, you can slalom through the columns). You can turn so long as you're not crossing lanes.
I look at current driver assistance systems as human-robot hybrid. Any driver assistance system should be required to be clear in it's intentions to the driver and give time to react, otherwise it's like a wild horse.
People who don't read manuals put themselves at risk.
Even simple systems like Cruise Control carry plenty of warnings that one should be familiar with when operating (from Ford's manual):
WARNING: Always pay close attention to changing road conditions when using adaptive cruise control. The system does not replace attentive driving. Failing to pay attention to the road may result in a crash, serious injury or death. WARNING: Adaptive cruise control may not detect stationary or slow moving vehicles below 6 mph (10 km/h). WARNING: Do not use adaptive cruise control on winding roads, in heavy traffic or when the road surface is slippery. This could result in loss of vehicle control, serious injury or death. [and few more, shortened for brevity]
EDIT - other people are pointing out the "no right turn" sign but I believe that's for cars to the left of the elevated tracks. The sign hanging by the traffic lights indicates the crossing road is one-way to the right, so that turn should be legal from where the Tesla is.
But it isn't planning a turn here so I don't think you can say "it would've made a tight turn and cleared the pedestrians." It looks like it has no idea where it's going and could have wanted to make a wide turn into the left lane, or even be swerving far out of its lane and then back in because it interpreted the intersection as something like a lane shift.
Not to mention the fact that it seems entirely legal for a company like Tesla to test such dangerous functionality on public streets with other drivers and pedestrians, with zero permission or regulation.
Agree this is a weird and very concerning bug.
> Not to mention the fact that it seems entirely legal for a company like Tesla to test such dangerous functionality on public streets with other drivers and pedestrians, with zero permission or regulation.
This driver seems to be someone who I would be ok with testing this. He's got his hand an inch from the steering wheel and obviously paying attention. I would rather this scenario happen, than Tesla be restricted to no public road access and they just unleash an untested system.
My fear is that a new version of the autopilot system could have new bugs introduced that could disrupt the navigation on a route that the user has confidence that the previous version of the autopilot could handle. They commute on route x every day and over time they've gained confidence in the autopilot on that route. This new update however is going to run them into some obstruction due to a bug. That obstruction might be those monorail pylons we see in the seattle video or it might be a crosswalk. A driver who had confidence in the autopilot might be well distracted having confidence in his automation and not be able to correct the situation before tragedy.
IMO autopilot should be banned until it can be proved to be 100% safe. I don't think we can get there until roads are outfitted with some kind of beaconing system that sensors in the car can read and cars on the road are potentially networked together cooperatively... and only then to be enabled on roads with those markers/beacons.
People in this thread deciding that the system is safe because it's no worse than a drunk driver or student driver are missing the point. We absorb those hazards into the system because they present only a handful agents in the system. Out of 100,000 drivers in a rush hour flow, how many are students and/or drunk.. probably very few. However as teslas keep selling with this feature, our new class of hazard agents keeps going up and up and up.
Hell we wouldn't even have to mandate it to be illegal at the political level. Perhaps the insurance industry will do it for us.
There is some level of reasonable safety that should be expected, but proving 100% safe isn't a realistic goal, nor is it even a standard for existing automobiles.
> This sort of close interplay between pedestrians and cars is very common in cities.
Yeah and I'm usually yelling at the idiot that wasn't looking where they were driving.
Except it usually happens when the pedestrian has already passed the area where you'd turn through or the intersection is enormous and the pedestrian just started crossing the other side.
I'd expect FSD to follow the law in this case though.
Not in the state in which this particular Tesla was being driven:
"The operator of an approaching vehicle shall stop and remain stopped to allow a pedestrian, bicycle, or personal delivery device to cross the roadway..."
"when the pedestrian, bicycle, or personal delivery device is upon or within one lane of the half of the roadway upon which the vehicle is traveling or onto which it is turning"
Interestingly enough that should make it legal to pass in front of pedestrians who are crossing traffic going the opposite direction, provided that you do not interfere with their right-of-way.
We know that pedestrians were far enough not to feel threatened as they waved back responding to drivers wave. Police was at the intersection also and they did not look concerned either.
Whatever the hypothetical (it certainly might have missed the pedestrian, or made a last second emergency stop) - it suddenly veered towards a pedestrian that was already walking on the road, and that alone is absolutely not OK. Scaring the living daylights out of people isn't acceptable, even if you might not have killed them without an intervention. And let's be fair, if the pedestrian had not paid attention, and the driver neither, this certainly could have lead to an accident. Even if it were "just" an unexpected emergency stop; that itself isn't without risk.
"Half of the roadway" is defined as all traffic lanes carrying traffic in one direction. In the case of a one-way road such as the one in the video "half of the roadway" is the whole road.
In your 4-lane hypothetical, if there are two lanes in each direction you can drive through the crosswalk if the pedestrian is in the farthest away opposite direction lane from the lane you are in. In a 4-lane road with 1 lane in one direction and 3 in the other, you can drive through if you are in that 1 and the pedestrian is in the farthest 2 away from you. If you are in one of the 3 going the other way, you have to stop no matter which lane they are in, because they are either in your half (the 3 lanes going your direction), or within 1 lane of your half.
If a driver is trying to turn right, and the pedestrian has just entered the crosswalk other side of the street, 40 ft away, it seems stupid to wait for them to clear the whole 40 ft when your right turn doesn't put them at risk. I say that even when I'm the pedestrian in that situation.[1]
If the worst thing about SDCs that they do these kinds of violations (which would also include going 62 in a 60 mph zone), then I would say SDCs are a solved problem.. Though to be clear, Teslas are not at that point!
[1] Not just idle talk -- I cross East Riverside Drive in Austin a lot on foot, where this exact situation is common.
Apparently, it's not what the pedestrian thought.
As a motorcyclist, runner, and cyclist, I can tell you that drivers will look you right in the eye as they pull out in front of you.
we make eye contact and look for acknowledgement or an indication of intent.
That behavior stands a good chance of getting one killed or hurt, especially in Seattle, the land of "oh, no, you go first..." Look at the wheels, not the eyes. The wheels don't lie.
I've been mostly looking at the OSD---it doesn't look like the car noticed the pedestrians until after the driver had taken manual control.
Tesla FSD Beta 10 veers into pedestrians with illegal right turn - https://news.ycombinator.com/item?id=28545010 - Sept 2021 (118 comments)
Less recent and less related:
Watch a Tesla Fail an Automatic Braking Test and Vaporize a Robot Pedestrian - https://news.ycombinator.com/item?id=24588795 - Sept 2020 (111 comments)
To Downvoters: Well it is actually true. [0] and just recently another crash involving an emergency vehicle. [1]
So there is a strange habit with Tesla FSD and red objects in its view. Given those incidents, care to explain why I am wrong?
[0] https://www.autoblog.com/2018/01/23/tesla-autopilot-crash-fi...
[1] https://www.reuters.com/business/autos-transportation/us-ide...
Because even in cases where cars have hit emergency vehicles, it's because the software didn't see the vehicles at all and just continued driving straight in its lane. Whatever flaws the Tesla vision system may have, the idea that it is programmed to deliberately seek out and crash into emergency vehicles seems pretty far-fetched (much less that it would mistake a person wearing a red coat for an emergency vehicle and therefore attempt to crash into it); I assume this is why people are downvoting you.
One of the big limits of this kind of AI is that it does not provide human-legible explanations for its actions. You cannot put it in front of a tribunal.
So, the same / similar data fed to the same / similar algorithms and the state of the code examined by qualified experts (programmers).
Whether that kind of logging is always on or was specifically on here, I don't know, but I'd expect Tesla can analyze why this happened: the car does have the ability to explain itself; it's just that owners and drivers do not have access to that explanation.
They can use those intermediates to project a kind of thought process others can look at - and you've probably seen videos and images of that kind of thing too; i.e. a rendered version of the 3d intermediate world it's perceived, enhanced with labels, enhanced with motion vectors, cut into objects, classified by type of surface, perhaps even including projections of likely future intent of the various actors, etc.
Sure, you can't fully understand how each individual perceptron contributes to the whole, but you can understand why the car suddenly veered right, what it's planned route was, what it thought other traffic participants were about to do, which obstacles it saw, whether it noticed the pedestrians, which traffic rules it was aware of, whether it noticed the traffic lights (and which ones) and how much time it thought remained etc.
...at least, sometimes; I don't know anybody working at Tesla specifically.
Here, for example waymo has a public PR piece kind of highlighting all the kind of stuff they can extract from the black box: https://blog.waymo.com/2021/08/MostExperiencedUrbanDriver.ht...
And while they emphasize their lidar tech, I bet Tesla's team, while using different sensors, also has somewhat similarly complex - and inspectable - intermediate representations.
In the days when Sussman was a novice, Minsky once came to him as he sat hacking at the PDP-6.
"What are you doing?", asked Minsky.
"I am training a randomly wired neural net to play Tic-tac-toe", Sussman replied.
"Why is the net wired randomly?", asked Minsky.
"I do not want it to have any preconceptions of how to play", Sussman said.
Minsky then shut his eyes.
"Why do you close your eyes?" Sussman asked his teacher.
"So that the room will be empty."
At that moment, Sussman was enlightened.
(https://news.ycombinator.com/item?id=10970937)
IIRC, in the incident where the Tesla [Edit: Uber self driving car] collided with a pedestrian pushing a bicycle in Arizona, the Tesla repeatedly switched between calling the input a pedestrian and a bicycle. And took no evasive actions while it was trying to decide.
That was Uber's self driving car program. Notably, the SUV they were using has had pedestrian detecting auto-stopping for several years, though I'm sure it's not 100%
That particular kind of black box is very black; it has hundreds of thousands to millions of inputs feeding a hyperdimensional statistical model.
ed: modeless 1 hour ago
>That sign applies only to the lanes to the left of the pillars. It is legal to turn right there from the right lane. I've done it myself. Yes, it is confusing.
Not exactly the FSD robot-taxi experience the Tesla fans were promised. Instead, they got happily mislead with beta software.
I was being dumb and am wrong. This is YouTube's internal system for copyrights not DCMA.
Apologies.
Old comment below...
You can read up on how to submit DCMA takedowns for YouTube here -> https://support.google.com/youtube/answer/2807622?hl=en
The original uploader of the video submitted the DCMA takedown request.
YouTube's internal copyright takedown system is just that: Internal. DMCA is a legal avenue that you can use against YouTube and their users, but it wouldn't be via their internal copyright take down system.
It is in YouTube's best interests for you NOT to use the DMCA as there are avenues where it can leave YouTube themselves civilly liable (OCILLA[0]), so they work hard to funnel people into their internal copyright system that doesn't expose YT.
[0] https://en.wikipedia.org/wiki/Online_Copyright_Infringement_...
Edited the comment to note my error.
I imagine they probably started getting a lot of negative attention.
However it is supposed to be from the copyright owner [17 USC 512(c)(3)(A)(i)] and if it isn't, it can be ignored [17 USC 512(c)(3)(B)] and makes the person who does it liable for damages [17 USC 512(f)(1)]. Although that's probably pretty hard to use in practice.
Edit: It's about the copy on Twitter, https://twitter.com/TaylorOgan/status/1438141148816609285
https://i.stack.imgur.com/2KymN.png
Context - this is after the sudden sharp turn, as the driver is taking over.
- We can see the system is still driving itself; the driver has not taken over and is not a factor. The blue steering wheel shows this.
- The dotted line coming off the car shows its intended path. If you step through the video, you can see it decide it's going to make the turn (for some unknown reason).
- Before the driver takes over, as this frame shows, we see the car decide to turn back to the left and resume its correct course. In fact, if you step through the video, you can see the car is turning the wheel to the left, rotating under the driver's hands before the driver applies enough force to disengage.
- The car was aware of the pedestrian at the time it decided to make the turn (you can see them on the screen), and its clear the path would not have collided with the person even if the driver had not taken over, and the car had not corrected itself.
- But the car did correct itself, and intended to resume its straight course before the driver took over, as that frame above shows.
This is not "Tesla almost hits pedestrian!", this is "Tesla makes bad navigation decision, driver takes over before car would have safely corrected"
You can't prove something like that, I mean consider the irony that you typed this statement right after
> It certainly shouldn't have made this maneuver at all
How can you prove that it wouldn't have made another unexpected maneuver?
It's all speculation, and that's OK. I wish OP used a different title, because everyone seems to be getting hung up on it, when the lede is certainly the FSD being unable to maintain a straight line.
It's actually the interference of the driver that stops the turn halfway, and steers towards the people at the left starting to cross that street.
I wonder what would have happened when the driver didn't interfere. I guess that the right turn would have been completed without any problem.
But maybe I'm missing some extra info here?
Besides, the pedestrians were already on the road when the car initiated the turn. Note the "one way" sign: that street only has a single lane, and that lane was already occupied. It should definitely have yielded.
The right turn would (at best) be completed by making pedestrians wait for the car, which is backwards.
You’re supposed to wait for them to fully complete their crossing, and yes: the overwhelming majority of drivers here abide by that requirement.
I drove in US, and I know how slow and relax the traffic is. Come drive in Brussels or any of the big European cities (like Paris) and see for yourself.
Good to know US is civilized and EU is not. The difference is that you just have way more space than us.
Erratic navigation and driving. 1 second earlier the car was heading straight. The car placement was firmly to the left edge of its lane. A safe driver would slow down towards the turn and use the right hand side of the road - pedestrians will recognize this "body language" of the car and understand the situation better.
In our peripheral vision, the car placement and "body language" is probably more important than ostensible signals such as the turn signal (!)
This kind of "body language" is something we learn when driving and automatic drivers should adhere to it too, so that they can be predictable and interpretable drivers.
This maintains a different norm for the city about jaywalking than say Boston, so in principle the issue seems quite solvable.
E.g.,
git discussion on HN: of course we all know how git leverages merkel trees to make it workable to produce millions-of-line ambitious projects across timezones with people we've never even met before (much less are willing to fool around giving repo permissions to). But goddammit it takes more than five minutes to learn the feature branch flow. That's unacceptable!
self-driving car discussion on HN: I've seen plenty of human psychopaths nearly mow down a pedestrian at an intersection so how is software doing the same thing any worse?
It seems like it thinks the lane shifted to the right because it had to reason to do a right turn.
Anyone have more insight into what occurred?
If it is just the car, it looks like it was trying to turn and didn't consider the people in the crosswalk.
Edit: thanks for the explanation.
I'm guessing if he is videoing it and hovering over the wheel that much this has happened before and he is nervously expecting it?
https://www.sae.org/binaries/content/gallery/cm/articles/pre...
This FSD beta seems way too cocky and at the same time confused underneath, I think we will see quite a few incidents with this one unless everyones a good boy and is really in the loop with taking over quickly.
https://news.ycombinator.com/item?id=28545010
It was on the front page and then a few minutes later not on the first 5+ pages.
For one they do not actual approximate human eyes since our eyes themselves are gimbaled in our skulls and our skull is gimbaled on our necks. Our eyes can switch their focus very quickly and return to scanning very easily.
Not only are we processing the visual information but also feedback from all of our body positioning and other sensory inputs. Driving heavily relies on our proprioception.
Fully autonomous cars need a lot of different sensors and processing power. Even driver assist should be using a lot of sensors. Autonomous driving needs more and better sensors than humans because it needs to deal with humans that are both smarter and more irrational than the automation. Besides conscious abilities humans have a lot of inherent self-preservation instincts that might be locally optimum for survival but globally poor choices.
Tesla scrimping on sensors because their boss is an egomaniac is the height of hubris. A web service moving fast and breaking things and choosing cheap but redundant options is fine. Usually the worst that happens is people lose money or some data. Moving fast and breaking things in a car is literally life and death. Musk's hubris around FSD is sociopathic.
Put a human in a room with no shadows, unrecognizable objects, and misleadingly sized objects (tiny cars, huge animals), and then watch them fail to judge distances or other vision tasks.
"Video accusing Tesla FSD beta of almost hitting pedestrian receives DMCA take-down."
On the DMCA takedown - that's pretty sketchy.
video of company dumping toxic waste into the ocena? DMCA takedown !
First it was big oil, then it was shortsellers, but I found those narratives sort of died out.
Who are they beefing with right now?
Level 2 automated driving is defined as systems that provide steering and brake/acceleration support, as well as lane centering and adaptive cruise control.
So it is the same as in most modern cars (other brands also have 'environment awareness') but Tesla still market it as 'sit back and close your eyes' driving.
>calls it full self driving
What a joke & GREAT way to mislead customers.
One of the attributes of human perception is that it is TERRIBLE at maintaining persistent vigilance without engagement.
Even at a very low level, the nervous system is designed to habituate to constant stimuli; e.g., when you first encounter something that smells (good or bad) it can be overwhelming, but after a few minutes the same smell barely registers. More on point, spend some time looking forward at speed, or rotating (e.g., in a swivel chair), then stop quickly, and watch how your visual system creates the illusion of everything flowing in the opposite direction.
Now, scale that up to higher levels of cognition. The more the car gets right, the worse will be the human's attention. When a car here does almost everything right, people can and will literally read or fall asleep at the wheel. Until that one critical failure.
As a former licensed road racing driver and champion, I find the idea of anything between L2 and L4 to be terrifying. I can and have handled many very tricky and emergency situations at a wide range of speeds on everything from dry roads to wet ice (on and off the track) — when my attention was fully focused on the road, the situation, my grip levels, the balance of the car, etc.
The idea of being largely unfocused while the car does almost everything, then having an alert and having to regain, in fractions of a second, full orientation to everything I need to know then take action, is terrifying. 60 mph is 88 feet per second. Even a quick reaction where I've squandered only a half second figuring out what to do is the difference between avoiding or stopping before an obstacle, and blowing ~50' past it, or over it, at speed.
Attempts to say "it's just fine because a human is in the loop (and ultimately responsible)" are just bullsh*t and evading responsibility, even if human beta testing is fantastic for gathering massive amounts of data to analyze.
Among high risk and speed sports, it is almost axiomatic for us to draw distinctions between "smart crazy" vs "dumb crazy", and everyone knows the difference without a definition. The best definition I heard was that it's the difference between [using knowledge, technology, and skill to make a hazardous activity reliably safe] vs [getting away with something]. You can 'get away' with Russian Roulette 5 out of six times, and you'll probably get a great adrenaline rush, but you can't expect to do so for long.
Although this kind of "full self driving" has much better odds vs Russian Roulette, it is still unreliable, and the system of expecting the human to always be able to detect, orient, and respond in time to the car's errors is systematically unsafe. You will 'get away with it' a lot, and there will even be times when the system catches things the humans won't.
But to place the entire "legal and moral responsibility" on the human to 100% reliably operate a system that is specifically designed against human capabilities is wrong, unless you want to say that this is a [no human should every operate under these conditions], like drunk driving, and outlaw the system and the action of operating it.
[1] https://www.motortrend.com/features/how-much-are-tesla-insur...
Not necessarily more, but we do see some really horrifying ones that humans would rarely do. E.g., the car in FL that just full-self-drove at full speed straight under a semi-trailer turning across the road, decapitating the driver, or the Apple exec that died piling into the construction divider on the highway because the Tesla failed to understand the temporary marks on the road.
I'm fine with Tesla or other companies using and even testing automated driving systems on public roads (within reason). Ultimately, it should be better, and probably is already better than the average human.
My objection is ONLY to the idea that the human driver should be considered 100% morally & legally responsible for any action of the car.
Aside from the fact that the code is secret and proprietary, and even it's authors often cannot immediately see why the car took some action, the considerations of actual human performance make such responsibility a preposterous proposition.
The maker of the autonomous system, and its user, must share responsibility for the actions of the car. When there is a disaster, it will, and should come down to a case-by-case examination of the actual details of the incident. Did the driver ask the car/system to do something beyond it's capabilities, or was s/he unreasonably negligent? Or, did the system do something surprising and unexpected?
In the present situation, where the car started to make a sharp & surprising turn and almost ran over a pedestrian, it was Pure Dumb Luck that the driver was so attentive and caught it in time. If he'd been just a bit slower and the pedestrian was hit, I would place this blame 90%++ on Tesla, not the driver (given only the video). OTOH, there are many other cases where the driver tries to falsely blame Tesla.
We just can't A Priori declare one or the other always at fault.
It doesn't look like it would have hit anyone to me (it was only partway through turning, so was facing people when it or he aborted it, but that's not the path shown on the display), but it was definitely an illegal move.
Not that the driver was wrong to do what he did; we have the advantage of frame-by-frame replay. But the frame-by-frame does show that this is not what it appears at first glance.
Here's one: https://www.reuters.com/business/autos-transportation/us-pro...
Here's another: https://www.nytimes.com/2021/08/17/business/tesla-autopilot-...
Oh, another one: https://www.forbes.com/sites/lanceeliot/2020/05/16/lawsuit-a...
This appears to be the first one, in 2018: https://www.theguardian.com/technology/2018/mar/19/uber-self...
I'd keep going but maybe you get the point.
The car aborted (or was aborted by the driver) and stopped before finishing it's turn on the road from what I saw, leaving it at about a 45 degree angle instead of the 90 degrees it would eventually reach if it continued turning.
Was it though? Look closely, and maybe replay it a few times (I just did, another response to a different comment of mine had made this point so I looked). The car seems to have recovered itself, and the driver grabbed it a second later after it was already correcting.
That it decided it needed to turn all of a sudden us perplexing, and something that should be looked into, but it also went back to a straight path a moment later, either because of pedestrians too close or because of some other reason.
> nobody is arguing this error was so bad it could have killed civilians even if the driver was responsible.
"drove directly through where the people were going to be" most definitely implies that to my eyes. People die for exactly that reason all the time. Maybe I'm confused as to what you're trying to communicate.
> The driver took over way before the car entered the crosswalk section.
+
> "Entered the crosswalk section" is a rather euphemistic way of describing "drove directly through where the people were going to be" reply
=
"The driver took over way before the car drove directly through where the people were going to be."
If you are right that the car stopped itself then it's certainly not as large an error as I had assumed. I think what I was mainly trying to communicate was simply a disagreement with the person I was responding to who implied there was little wrong with this behaviour at all.
You are required, by law, to stop for pedestrians in crosswalks. Every intersection in Washington state is a crosswalk, marked or not.
But if you were an observer on the corner and could not see whether it was a human or an AI was driving the car, you would almost certainly consider whatever the car did (starting the turn) while pedestrians were actively crossing dangerous.