Hacker News Reader

Comment by retrac | Hacker News Reader

I'm slightly alarmed by the general scenario of a driverless vehicle approaching an emergency situation. A human-driven vehicle will respond to gestures from firefighters to go around, etc. And even most dull humans will have more situational awareness and flexibility than an AI is likely to, particularly as these sorts of scenarios are edge cases by definition. E.g. depending on your jurisdiction and the situation, it may be proper to run a red light to make way for an emergency vehicle. The insistence of the driver waving frantically at you is part of that judgement call. Software is not ready for that yet, I think.

View on HN

driverless cars will have endless problems.

When I was on sales trip to KL in Malaysia there were times when the tropical rain could half flood a street in minutes. Our driver was able to creep around the shallow side of the road (alternating with cars in the other direction).

How would a driverless car cope? Would it even detect that there was problem? and if it detected it and wouldn't drive on that might be the worst thing to do if the flooding was going to get worse.

There's a one-lane bridge near where I live that would pose a similar challenge. Drivers coming from the east always have right-of-way, but the only way to know this is by reading the bespoke traffic sign posted next to the bridge. To make matters worse, during times of heavy traffic westbound drivers will often stop and let through a stream of eastbound traffic. How many cars are let through is negotiated with hand signals or flashing lights and smiles.

We will get the worst of both worlds, the car will tell you to take over in dangerous or confusing conditions but the person taking over will have minimal driving experience, on account of the car taking it all.

Driverless is a huge waste. Most of Tesla's bad press (the company/product, not Elon) is due to Autopilot and/or FSD.

> Software is not ready for that yet, I think.

It will never happen. Robots will never have the intelligence to manage dangerous situations in the open world, only in controlled environments. After a series of accidents passes the unforgivable threshold, certain levels of autonomous driving will be banned completely.

Maybe I'm too optimistic about self driving but it seems that a fail-safe default of "upon encountering an active emergency situation, stop at a reasonable distance away" is good enough and better than what a significant minority of human drivers do.

Who are the significant minority?

This is the same claim always given by the A.I boosters. Who sees firefighters fighting a fire and decides to just plow through and make matters worse?

Because by "significant minority" what you are saying is that 2 or 3 out of 10 human drivers do this.

Exactly. The fact that some humans are drunks or psychopaths does not mean that AI mistakes are going to just be accepted. We send people to prison when they make those kinds of mistakes, what do we do when machine learning models kill while functioning entirely normally?

Send a low level engineer to prison, of course.

That's actually not as wild as you think, because that's what happens when a building or bridge falls. The contractor that built the bridge, just like the company that designed it, the agency responsible for inspections and maintenance, etc, is also liable for damages, including prison if it's deemed so by the courts.

Of course, software engineers get treated with kid's gloves, otherwise, everyone just freaks out and cries and cries, because their robotic utopia dreams are meaningless nonsense.

Apparently, smart enough to code, but not smart enough to take responsibility for their actions.

> We send people to prison when they make those kinds of mistakes, what do we do when machine learning models kill while functioning entirely normally?

Assess a massive fine. Money is lifeblood of a corpo.

How do we know we can make the machines smart enough to detect the dangerous situations it is incapable of handling on its own?

It's just a bit too perfect, no?

How do we know we can make humans do it?

If the dangerous situation is a flash flood or landslide stopping may be entirely the wrong response. The entire problem with these rigid failsafes and the whole idea of ai-driven cars is this idea that a programmer can predict every situation the car will encounter by extrapolating from the climate and road conditions in places like san francisco.

One project I worked on was a safety critical system that almost got sunk by an overly-simplistic fault response. The device would reboot if it detected any sensor readings out of range. One day someone put a bag over it while it was delivering concentrated oxygen from the air and it rebooted because the ambient o2 sensor was reading well below what anyone could reasonably expect were the machine not enclosed. The patient had to be bagged while the machine came back on.

You simply can’t predict every emergency situation and you most certainly shouldn’t assume a simplistic single-response model is enough to get the passenger to safety. Humans are really good at inferring correct behavior socially and it would be much better for an AV to watch what other drivers are doing to figure out how to handle unexpected situations. To the degree an AI-driven system is unable to do this it is completely unsuited for use on the road.

I think many of us aren’t used to working on anything actually critical to human lives.

And I wouldn’t be surprised if quite a few of those who are have never actually fully understood what that means.

I certainly don’t.

But then, there’s a reason I stay away from life-critical systems.

What about a dangerous situation like a criminal trying to enter the vehicle where the right response is to avoid them and drive away?

I think never is a strong word. You genuine believe that we'll never create an equivalent or even comparable intelligence to our own artificially?

I think there's no stage at which you trust such an entity with something like driving until it wants a robot to drive it's car so it can do other things than focus on driving. In the same way you don't trust a 5 year old with a loaded gun, you don't trust real synthetic intelligence with something like driving you safely. Not until it reaches the level where it wants equal rights, suffrage, and generally to be treated as alive. That's way off into science fiction land.

I don't get why people think intelligence requires things like a desire for suffrage...

There is no particular reason why advanced planning, modelling and pattern matching ability should need human-like goals.

Heck, even counting votes would be quite impossible (how much divergence does a copy need before it gets its own vote? What it not all modules/nodes are copied?).

Also, the anti-suffrage women seem to have been erased from history as far as popular discourse goes... guess the Women's National Anti-Suffrage League isn't taught in schools (support amongst women for extending the vote was very lukewarm, becoming a majority only after the law changed).

There is no particular reason why advanced planning, modelling and pattern matching ability should need human-like goals.

Because driving requires the ability to understand what other drivers are doing. It requires a theory of mind, which means understanding human goals.

Moving from point A to point B is the goal of humans while they drive. You don't need "human like" understanding of human goals. You do need to account for the stupidity of people though.

Human drivers communicate with each other through subtle cues, body language, etc. Moreover, pedestrians, children playing in the street, pets running onto the road chasing a ball, construction sites, parades, protests, are all things involving intelligence and theory of mind that you can't reduce to "the goal is to point A to point B."

We can try to reduce these things. Most things can be reduced to "obstacles". If obstacles are behaving the way we predict, then they are logical, otherwise they are flagged as illogical. When i engage in "subtle cues" or "body language" while on traffic i never trust it. If someone starts to drive illogically i never try to predict what they will do next, i just flag them as illogical and prepare for the worst.

There is a way to solve all these issues: keep humans (and other animals) away. This is how it's done with railroads. Unfortunately, there's not a whole of advantage in eliminating the engineer of a train since the ratio of engineers to passengers is so small to begin with.

I think instead of flirting with this absurd fantasy of self-driving cars we should be doing more to invest in railroads and other forms of mass transit so that people don't need to drive at all. Europe and Asia are far ahead of North America in this regard.

Mass transit doesn't get you exactly where you need to go. In areas with lower population density, it's especially unprofitable and difficult to implement -- and even if communities can afford it, it will deliver people even FARTHER from where they need to go.

I'm a transit advocate, but there are some fundamental issues that need to be resolved first, especially in North America.

Even in Europe, there are a lot of places you can't get to or get to easily without a car. I've done long distance walks in England, for example, and once you get beyond cities and larger towns, it's often the case that there may be a once or twice a day bus.

Not true, at least for any skilled driver. N=1, & my prior racing training & experience is an advantageous influence, but I am constantly assessing the intent, attitude, skills, etc. of other drivers to adjust my actions.

Of course it is trivially true that we all intend to go from point A to B. But that is almost irrelevant. It is the million micro-decisions on the trip that count.

Without seeing the human, I assess in seconds whether the other driver is unusually high or low-skills, focused or distracted or impaired, polite or rude... and this constant assessment, with human understanding, of (others') human goals and attitudes is a key element of driving skills, and yes this is basically all covered in your exception of needing to account for stupidity of people. That exception is just bigger than you think.

> It requires a theory of mind, which means understanding human goals.

Like chess?

Chess doesn't require understanding human goals if you're an AI, just running through the space of winning moves that your training encountered. An AI doesn't think "ah, rook to E4, knowing Kasparov, he will then castle and I can do X". It'll just go "lol I encountered 3 486 864 games with this particular board pattern and 49 652 lead to a victory lemme pick one for the lulz".

Chess is really just about searching through as much of the space as possible, and you can take a few seconds. And believe me, there's a whole lot more stupid behaviours you can encounter on the road while you have to react in less than a second.

> It'll just go "lol I encountered 3 486 864 games with this particular board pattern and 49 652 lead to a victory lemme pick one for the lulz".

That isn't really an accurate description of how the modern DL bots work. They don't need to reference any database of past games while they play.

Chess AIs don't need to develop an understanding of the goal, they're hard-wired with the rules of the game and its goal-states.

As an experiment, try making a chess AI without explicitly giving it the rules of the game and see how it performs.

I think that's already been done: https://en.wikipedia.org/wiki/MuZero

Almost, but not quite, exactly unlike chess.

So not like chess, but like how chess was supposed to be?

Chess is a perfect information game. Both opponents can see the entire board at all times. Reality, nor even driving, is not anywhere close to that. Weather, time of day, climate, seasonality, and other drivers throw constant and voluminous amounts of unpredictability into the mix.

> Reality, nor even driving, is not anywhere close to that.

This reminds me of the kinds of intuitions people had about Chess and Go. In retrospect they seem silly, but it made plenty of sense to them at the time. The fact was that there was a solution that machines could use that humans couldn't use. Naturally, a solution that humans couldn't use was hard for them to anticipate being effective.

You can understand something/someone without agreeing with them and having the same goals.

Well, "never" is a strong enough word to extend even into science fiction land (or 100, 1000, 10000, 100000 years into the future and beyond). Hence the reaction you got.

Or until we are able to change the nature of driving to make it require less creativity. As a simple examples, I suspect a road network where all cars were self-driving would probably be an easier software problem.

If Only.

Single-track railroad like in airport terminal-to-terminal transport? Fine.

Out on the open rail network? Not even close. And that is a LOT more constrained & predictable than an open road network.

Getting to all cars being software-driven seems as much in the near-mid future as was the idea of having full vehicle-to-vehicle telemetry &communications - i.e., sounded like a great idea until it crashed on the rocks of practicality (and inter-corporate politics?); not likely happening in a relevant time frame.

I personally don’t. We are not guaranteed endless progress or a good future.

> endless progress or a good future

The optimist believes progress will halt at a convenient moment. The pessimist fears otherwise.

Even if one didn't accept that AGI will eventually be developed, I agree with you that "never" is too strong. Computers perform many tasks now that were once unthinkable, and I'm pretty confident they'll eventually drive cars well. Whether that time is remotely close however is another question.

It will definitely happen, and likely (>50%) within the next 10-15 years.

We need true GAI to do this, of course, and the question is: will a "real" GAI be willing to drive a car? How much will we have to pay it?

GAI is a pipe dream. There is no technology that we can even conceive—let alone build in the next 10-15 years—of which passes as GAI. Our current AI is based on statistical inference, GAI will need to be able to do far more than infer from data and interpolate actions.

No we will never have GAI.

---

I know never is a long time, but I can comfortably predict this because there is not only technological problems with GAI, but also philosophical problems. The way we define intelligence is such that if we find or invent something that combats human intelligence we immediately redefine it to exclude that aspect.

The question of if an AI can be regarded as a sentient being has already been debated ad nauseam in science fiction (e.g. Data in Star Trek).

All that's needed is someone to achieve GAI at a supercomputer level. From then on, it's just a matter of scaling it down. Your iPhone has more compute power than a supercomputer 30 years ago, after all.

And what's also a way to "cheat" is fusing computers and organic matter. Brain-in-a-vat style, or, for yet another Star Trek reference, Voyager's organic compute network (which IIRC even suffered a pathogen infection at one point). And that is definitely possible in the next 10-15 years. Give Musk's Neuralink enough apes to euthanize and it will get done.

Your iPhone has more compute power than a supercomputer 30 years ago, after all.

In 30 years we will not have an iPhone-sized device with the power of today's supercomputers. Transistor scaling is in its final stages. We'll be lucky to get 2-3 more process nodes and that'll be it.

This assumes we don't use anything but transistor scaling for the next 30 years in mobile computing.

Even if we find a novel semiconductor that can scale 1000x better then silicon transistors (say DNA semiconductors), then we are still limited by classical computing. Meaning our most intelligent systems are still just good old statistical inference (I say still, but statistical inference is really powerful, and getting 1000s of orders of magnitude better at statistical inference means a world of amazing discoveries).

Perhaps quantum computing will enable new kinds of inference, then we will need to scale superconductors like we did transistors in the past 75 years, that is possible (not in the next 10-15 years though; 30-40 years is more realistic). However I don’t think this will lead to anything more then amazing statistical inference. People theorized artificial neural networks way back in the 1940s and we already had algorithms for it in the 1970s. I’m not aware of any algorithms, not even theoretical ones that gives us AGI, on any computing device (not even quantum computers).

GAI is not limited by computing power, but theory. Yes we will be able to make better inference by feeding more data into our models, supervised learning will be able to work with thousands of parameters, etc. This will lead to all sorts of new discoveries in many fields, including biochemistry, astrophysics, neurology, etc. It will also bring us a bunch of new fun toys to play with including better and more capable robots, and it will also lead to better engineered social systems (including traffic engineering; but also public transit, and—yes—self driving vehicles—but we already have self driving trains, so it is not as big of a deal as you think it is).

However this misses my point about AGI. It is not about computing capabilities but theory. AGI will have to evaluate situations that has never arisen before without having data that directly directs it. I don’t see how statistical inference can do that, and as such we will need an entirely new methods of training it. Like I said, we haven’t even conceived how an AGI technology works, there are no algorithms which will work in theory even with unlimited computing power. Contrast this, for example, with quantum computing, which has a bunch of algorithms demonstrating it’s capabilities. With AGI all we have is “perhaps our cars become better at driving them selves”. I’m not buying it.

I'm trying to understand the worldview that leads to what you believe. As far as I can tell, you're assuming there is a supernatural aspect to the human mind.

According to my worldview, humans are just robots built out of DNA, proteins, etc. The idea that machines couldn't do everything a human can do (and more) doesn't make any sense to me.

I’m sorry but I don’t have that world view. I must not have been clear enough (english is not my first language; and I’m not even super clear in my native language).

I actually don’t believe in the superiority of the human mind, I think the word intelligence is actually a scientific distraction. Humans are remarkably—but not uniquely—adapted at communicating with each others and manipulating our environment for our benefit (most of the time; but sometimes to our determent, e.g. by causing a climate disaster). My statement about intelligence was not to share my believe about humanity, but a statement about human believes about our humanity (a theory of mind of sorts). I believe that humans are so human centric that we will always evaluate intelligence in human terms. As such nothing will ever supersede it as we will simply alter our former believes in order to keep our mind at the center of the universe.

Now as for machines that can do everything that humans can do (and more), I still consider this a pipe dream, even if we disregard the word intelligence. This is not because I believe the human mind is superior to machine, it is more because we haven’t conceived of how we would do this. Statistical inference has its limits, even with all the computing power in the world, it will never do what specialized machines like our brains have evolved over millions of years to do (well maybe if we allow our machines to freely evolve and procreate and set them loose in a dynamic self sustaining environment; but that wont provide any benefits for us).

I also have my doubts about the utility of AGI. People state that full self driving cars is easier with AGI, but I haven’t seen any proof of this statement (nor for the utility of AGI driven cars for that matter; we’ve had self driving trains for decades without AGI). With traditional machine learning models, there is tons of utility in the data analysis in all sorts of fields (just like there was with factor analysis and linear regression before it), but with AGI I fail to see what we will get from a vague notion of can do everything humans can do, which doesn’t apply to traditional machines that are specifically programmed for that task. In short, I don’t even think there is a market for AGI.

I may have read too much into your original post. I think I get what you're saying now.

We apply the concept of intelligence to other species, so I don't see why we couldn't apply it to machines. If people want to redefine intelligence so that humans stay on top, then they are wrong to do so. There is undeniably an objective component to it, even if there are many cultural factors at play as well. If you don't like the word "intelligence", replace it with "competence", "productivity", "performance", etc. and the same argument will apply.

You raised two other issues: the possibility of AGI and the utility of AGI.

I think we have good reasons to believe that it's possible. Three possible future scenarios: (1) we get to the point that we can simulate brains to a high degree of accuracy in software, (2) we get to the point that we can build artificial brains out of neurons made of silicon, (3) we get to the point that synthetic biology is advanced enough that we can engineer new kinds of intelligent organisms. These might be science fiction by today's standards, but they show that there is no issue with conceivability. As an aside, the human brain runs on relatively low power consumption and operations per second, so computational power isn't a bottleneck. We have enough compute, we just don't have the right algorithms yet.

As for the utility of AGI, I strongly disagree that there is no market for it. By definition, an AGI can do anything that a human can do. There is already a job market for humans. As soon as AGI can compete with humans in domains like management, engineering, scientific research, education, etc. I don't see why there won't be a huge economic incentive to adopt it.

> As an aside, the human brain runs on relatively low power consumption and operations per second, so computational power isn't a bottleneck.

You hit the nail on the head here (pun intended). Brains (and other adoptive biochemical processes which reinforce behavior) are way more ingenious then to be described as a simple computer. It is not just neurons firing in patterns with synapses which can reinforce certain pathways. No, there is a whole biome in our bodies, from our guts, our muscles, through hormonal systems and yes, nervous system. Our behavior is also influenced by bacteria and other microorganisms that doesn’t even share our DNA (but sometimes actually interacts with our DNA). This complex machinery has evolved over millions of years, and I don’t think there is any way for us to simulate it. Even if we look outwards, our behavior is just as much influenced by each other as it is by our brains. So for a successful simulation you’d need to “raise” your machine in a similar manner, allowing interactions with a whole community (or more) that treats it as equal. Our brains are merely a tool which encodes this interaction as learned behavior. Just building a brain will miss all of that and give us a lousy model.

In statistics we learn that: “No models are correct, but some models are useful”. I’ve always taken this to mean that there is not always utility in a 100% accurate model. I think this applies for AGI. If we create a machine that can do everything that a human can, and better, what can we use it for? You claim increased competence, productivity, and performance, but we already have machines that do that. And we humans have found a way to work pretty well with those. We even have fully automatic vehicles that transport millions of people every day (e.g. in Vancouver B.C. and Copenhagen). It doesn’t need to be better then humans in everything, just accelerating and decelerating in the right places, it doesn’t need to know the airspeed velocity of an unladen swallow, it will only need to know if something is blocking the tracks. Traditional statistical inference will make these systems better and cheaper, and they will benefit humanity, there is no need for them to have generalized knowledge beyond their immediate utility, so they will never be given one.

I agree with you in the sense that I think many problems don't require general intelligence once you have a solution. However, to come to those solutions it took decades or centuries of intelligent humans working on those problems. The potential benefit of AGI isn't in automating the problems we already know how to solve (although it might help refine our solutions). It's in accelerating progress in domains where there are unsolved problems.

Sometimes we get hung up on comparing AI to humans. I wouldn't want an AI that does math and physics in the same way that humans do. Our brains aren't very good at those things. I'm more interested in the potential for AI to find new, more effective ways of working in these areas.

> In short, I don’t even think there is a market for AGI.

AGIs by their own general purpose nature can improve other AGIs, possibly including themselves. AGIs can improve other less general purpose AIs.

Either you could ask for the AGIs to improve what you need for you, or the AGIs might do it by themselves.

The simple difference is humans have the biological goal to replicate their genes. That is the root source of intelligence. Software does not because it has not evolved. If you want real AI then start with simulating life in a simulated environment as complex as the real world for many thousands or millions of generations.

Humans may be like robots but their programing is not statistical pattern matching or Boolean logic. Rather it is the intelligence selected for by evolution to achieve the goals of an individual organism.

How does this not make sense?

Intelligence does not just happen absent goal directed evolution.

How does this not make sense?

As far as I can tell, your points don't apply to what I said. I'll quote myself:

>The idea that machines couldn't do everything a human can do (and more) doesn't make any sense to me.

In other words, it doesn't make sense to me when I hear impossibility arguments about AGI that appeal to something special about humans. I'm not making a claim about software and hardware in 2023. I'm saying in principle, there is nothing stopping us from building artificial systems that are just as complex and capable as evolved biological organisms, other than our lack of knowledge of how to build such systems.

>Software does not because it has not evolved

Software does evolve. To get evolution you need three things: replication, mutation, and selection pressure. We replicate, mutate, and select software all the time. I don't think you need a large scale simulation to get evolutionary dynamics.

>Humans may be like robots but their programing is not statistical pattern matching or Boolean logic

That might be true, but I'm also skeptical. Part of the problem is we don't really understand intelligence very well. If we really understood what makes humans capable of intelligent behavior, we would be able to replicate it in artificial systems. I don't see a fundamental barrier other than having the right algorithms.

You might be thinking about evolutionary learning (or more broadly evolutionary algorithms), which is a machine learning method different from—the more common and more successful—artificial neural networks. Several implementations actually exist, and I think some are even quite successful, but none that I’m aware of have been shown to have AGI capabilities.

> the way we define intelligence is such that if we find or invent something that combats human intelligence we immediately redefine it to exclude that aspect.

And what makes you so sure the opposite won't happen? E.g. GAI will be achieved earlier by excluding some aspect of human intelligence that some people deem essential. The whole "problem" is that we can't agree on what constitutes intelligence. I predict that it won't be long before AI systems can score 200 on standard IQ tests.

>I predict that it won't be long before AI systems can score 200 on standard IQ tests.

I don't know of a "standard" iq test that goes past 160 or so.

I think your "of course" is glossing over too much. It is very, very far from a given to me that GAI is required in order to solve self-driving cars well. I think it's reasonably possible we'll have good self-driving cars within 15 years (50% sounds about right to me for the longer end of that time horizon), but I highly, highly doubt we'll have GAI in anywhere close to that timeframe.

Have you tried any of the state-of-the-art out there?

ChatGPT is easily circumvented to violate all its own guardrails with a few instructions.

Knowledge without understanding.

It clearly doesn't understand it's own rules well enough to realize when it is asked to violate the rule "don't do C" that being asked to do A->B->C is the same as being asked to do C.

I donno the robots in terminator seemed to have some self awareness. Why can’t we apply the same to cars?

Your comment, I am not sure if intentionally, encapsulates what is really going on with self driving cars. I.e. people assume fantasy or fiction can become reality with enough effort. This is not always true...

> I.e. people assume fantasy or fiction can become reality with enough effort. This is not always true...

More or less hit the nail on the head :)

People can't even manage dangerous situations all the time. They panic, they freeze, and they make mistakes.

Driverless cars don't need to work perfectly all the time. They just need to injure fewer people than human drivers.

Sounds like having support for interpreting human gestures will solve some of the problems. But how do we distinguish real emergency scenario from fake one?

Such kinds of problems will only scale up as we apply L5 cars take on more public roads, but I wonder how long or what changes will have to be done in current road signs, critical city infra to support them.

> But how do we distinguish real emergency scenario from fake one?

A car shouldn't make that call. It should err on the side of safety and assuming the emergency is real.

When you see flashing lights on the highway coming up behind you, do you think "I bet that ambulance is faking it so they can get home faster", or do you just move over?

But then you have essentially enabled everyone to wave your car to a stop, which could be very convenient for certain types of people.

I think the simple truth is that we are still a long long long time away from fully autonomous cars to be safely deployed anywhere except on very specific paths that are mostly closed off to everyone else.

But they could be tremendously useful there, for instance driving buses or airport shuttles and that kind of thing.

So this is currently possible today: I can walk out on any street today and redirect traffic. With or without my reflective vest, drivers will generally listen to me. And self-driving vehicles must as well.

Note that when I am doing traffic direction with appropriate authority, most drivers will never have heard of the organization I'm a part of (a local CERT team), nor have any real way to vet the authority I have. The authority they can determine I have is mostly that I'm doing it and no police officer has come by to tell me to stop doing it.

(Note that CERT members do not and should not self-deploy, mind you, and if you choose not to heed our directions, we almost certainly can find a cop car to come let you know you screwed up.)

Also bear in mind, many people directing traffic aren't public safety personnel at all. Construction workers are private employees, events have their own staff which direct traffic in various contexts, including public roads.

There's no definitive way to determine that someone is authorized to direct traffic, cars just need to obey them.

>There's no definitive way to determine that someone is authorized to direct traffic, cars just need to obey them.

Exactly! But there still is a way. You don't obey anyone in any circumstances because that would be dangerous. People have a feeling for it, computers don't yet.

I'd obey such traffic commands from just about anybody in plain street clothes, unless they looked obviously crazy or if I was in part of the world where car jacking is common. At the very least I'd stop and ask them what was up. Maybe there is some dangerous condition ahead and they're trying to warn me.

Does the computerized car understand the meaning of the words "Turn back because the bridge is out"?

I miswrote, my meaning wasn’t that I would stop for nobody, but that I wouldn’t stop for just anybody in just any situation.

Essentially the same that you said.

This is something that people are very good at judging but software isn’t.

I’m trying to say that until we have figured out AGI, autonomous cars driving among people are very dangerous.

I bet you’d eyeball it though, right? Like if someone was telling cars to stop, looked nervous, and had a hand in a large enough pocket to hold a gun, I might head in the direction they were sending people (depending on visibility I guess) but I’d probably not slow down.

Human drivers are smart, and will generally judge your appearance, confidence, and context to determine whether or not your actions are legitimate. And you likely always pass these tests.

Imagine a teenager, dressed in street wear, who jumps into the middle of a street to stop traffic in a way that has no explanation given the scene, while their friends laugh on the sidewalk as they observe.

Or a group of men in masks stopping a car in an alley as their accomplices surround the vehicle.

Any human driver will recognize these scenarios as illegitimate. It isn't just the act of directing traffic that is judged in isolation.

In basically every example you gave, you still have to stop until people move though. The result is the same, despite everything else. The only one that's a bit iffy is if you're being carjacked, is a bit of an exception.

Legitimacy isn't really the problem, it's the circumstance of a human being in front of your car that prevents you from moving forward, and beyond getting out of your car and talking to them, or just waiting, you don't have much of a choice in what options you have.

I agree with the intent of your point, but what's the alternative? You can't just keep driving if someone is standing in the middle of the road.

Those were two examples, but they're not exhaustive. Traffic direction can (and often does) happen without someone blocking the vehicle.

In the case of an illegitimate request, a human driver might question the request and take a different action instead.

Are you being directed to run a red light? If you judge that someone is maliciously asking you to do so, you probably will choose to remain still instead.

Are you being directed to stop? If you judge that someone is maliciously asking you to do so, you may choose to turn around, reverse, turn into a parking lot or otherwise remove your vehicle from the situation. You don't have to run people over to refuse a command to stop.

And if you were completely blocked from moving, you'd probably call the police.

My point is that "always follow hand signals" is trivially problematic. I would expect that if self-driving cars followed any hand signal they saw, you'd see kids doing it for the lulz on the sidewalk in front of their school. ..."traffic jam challenge". After all, someone is less likely to act maliciously towards a human than they are a machine.

Okay, so your point was more about the specific directions being given, than the presence of a person in the road. Overall I agree.

One reason carjacking is rare is because people don't foolishly stop for obvious carjackings. If an autonomous car has trouble telling the difference between a carjacker and a construction worker, then it will be much more likely to get carjacked.

My brother and his teenage friends redirected traffic into their school grounds to prove that they could do it. They also passed all 'sure, looks legit' tests.

Humans aren’t perfect, but their BS detectors are better than nothing.

The more important test scenarios are life threatening or unreasonably disruptive situations rather than harmless pranks.

PKE transmitters given to firefighters with some mechanism to cycle private keys on autodriving taxis weekly.

Nowhere near perfect, but that makes it a lot harder than just getting a spike strip or a hand gun or a demolition derby car. We’re talking about people willing to risk a decade in prison to “do criminal things to a car and/or passengers”. How much work do you think they want to put into disabling a car with a remote? Why not just wave a gun in the driver’s face at a stop light?

> Note that when I am doing traffic direction with appropriate authority, most drivers will never have heard of the organization I'm a part of

Reading between the lines it doesn’t seem like there’s a lot of communication to emergency services etc on how to deal with autonomous vehicles.

Even the police seem to be operating under misconceptions about what the vehicles can and can’t do.

Isn't it the vehicle operators operating the vehicles while ignoring that there are things that the vehicles must do in certain scenarios to be suitable for operation?

So there's no monolithic entity that handles public safety in any given area. Overlapping jurisdictions of police, fire, public works, utility companies, construction companies, volunteer orgs, etc. all will end up on the streets for different reasons.

If self-driving cars cannot cope with this, they should not be allowed on public roads. Very little has convinced me they should currently be street legal.

>most drivers will never have heard of the organization I'm a part of (a local CERT team)

Will you have to file a CVE after the incident?

As a sysadmin and a volunteer responder, I do find the fact CISA and FEMA each have a CERT program that are completely different things pretty funny.

Community Emergency Response Team: https://community.fema.gov/PreparednessCommunity/s/welcome-t...

Agreed - faking an emergency is highly frowned upon and I'm betting it's a crime in most places. There's even a similarity with the following aspect of the Geneva Convention, which is that pretending to surrender is so harmful to norms that it is literally a war crime. https://en.wikipedia.org/wiki/Perfidy

"In the context of war, perfidy is a form of deception in which one side promises to act in good faith (such as by raising a flag of truce) with the intention of breaking that promise ... Perfidy constitutes a breach of the laws of war and so is a war crime, as it degrades the protections and mutual restraints developed in the interest of all parties, combatants and civilians. "

Pretending to surrender is something that's shown a lot in Star Trek and similar shows, and Pirates of the Carribean, but in our real-life world of 2023 it's a UN war crime to pretend to surrender, and it's really not a nice thing to do because if some people sometimes pretend to surrender, then pretty soon no one will trust surrender attempts and people who wanted to stop fighting will needlessly be lost.

The idea of criminals faking a medical emergency is similar imho. If I were in a position of determining how much time robbers should serve in prison, I would personally recommend a much much longer sentence if they faked a medical emergency, for exactly this reason.

I saw a video showing perfidy committed by a Russian soldier in Ukraine. Several Russians were coming out of a building surrendering to some Ukrainian soldiers, when one of them came out with a gun and started shooting. All the Russians were killed in the subsequent firefight, because one of them ruined it for the rest. It may seem like a clever ruse in movies, but in real life it gets people killed.

It'd be funny if it wasn't so terrible. At this point it would be easier to list the war crimes not committed by the Russian army, they seem to be well intent on speed running the Geneva convention. One can only hope one day there's trials in e.g. The Hague to bring all war criminals to justice.

Russians just have to start working on advanced weapons and then they'll be excused from the tribunals like Unit 731.

> One can only hope one day there's trials in e.g. The Hague to bring all war criminals to justice.

I hope that as well, but I think neither the US, Israel nor China will stand idle next to a nuclear power getting dragged in front of the ICC - not with the numerous counts of crimes each of these countries could reasonably be charged with. The Israelis may be the only ones being able to get away with a lot of their stuff given that their opponents continuously enact crimes as well (e.g. hiding military command structures under hospitals and schools), but the US and China? They can't pull that card.

> It may seem like a clever ruse in movies, but in real life it gets people killed.

To be fair, the point of the ruse (in real wars; in heroic fiction this sometimes varies) is usually to get people killed, too; the question is just which people get killed.

It gets more people to die on ALL sides which is why it's considered a shitty thing to do, even if it's technically a "valid" tactic in some anarchic philosophies. The use of this ruse will cause people to stop accepting surrender which is BAD because it means that the people who die will be the average Joe and average Ivan in different countries who really don't want to kill anyone and just want to go back to their family and their life. These goodhearted people just want to go home and would gladly surrender. Imagine this thought experiment. I'll use your pen name dragonwriter as the name of a hypothetical super-human bandit / assassin / mercenary / gun for hire who sometimes does jobs like assassinations and Mission Impossible style burglaries of royal treasure. Imagine the DragonWriter bandit has no qualms about killing and so anyone who stands in his way gets shot with the intent to kill, AND now there are some rumors that a group of cops had DragonWriter surrounded and so he surrendered to them but then he pulled out a hidden sword and spun in a circle chopping them all in half. Now, any time DragonWriter tries to surrender in the future, his surrender won't be accepted, but what's even worse is: his action that bought him maybe a 5 second advantage (vs a million different other ways he could have won or fled the fight), and this tiny temporary advantage by the ruse has imposed a cost in exchange:

This one infamous sword fight has now poisoned the well with respect to the global norm. Next time there's some other totally unaffiliated hat-wearing super-powered bandit who's stealing the Ark of the Covenant or the royal diamond or something, if THAT bandit tries to surrender, there's a high chance that person might be killed, because their surrender is distrusted since DragonWriter changed the norm and now every fight between supers and cops is now on "shoot to kill, no prisoners, take no quarter" stakes. Imagine how many times a famous super like Batman would have been killed if the authorities never accept a surrender because the surrender norm is ruined. It can be repaired but that might take a decade or more to repair the trust that was broken.

With strong norms about surrender, red cross medics being protected, etc., we can have things like the miracle of the WW1 Christmas Truce which help the war to be a little bit little less of pure hell, for all the pure-hearted or indifferent people dragged into being a soldier, perhaps even drafted or press-ganged, to be the disposable cannon fodder for a bloody and disease and gas drenched war for no good reason more than rich politicians' egos.

https://en.wikipedia.org/wiki/Christmas_truce

These people just want to go back home to their wife or their mum and would gladly gladly surrender if given the chance, but that's not possible if someone ruins the norm! It's like, for example even if someone is a super-villain for example, they should really not take human lives of innocent people, or destroy irreplaceable cultural heritage sites, or damage/burn an ecological area like a rain forest, because those, once lost, are forever gone, and leave our world impoverished.

The number of criminals faking medical emergencies to avoid having to appear in court is pretty high. And of course three days after they're right as rain. It's hard to act against because of the risk of punishing someone that really needs it.

Faking your own illness is very different to faking an ambulance with lights and sirens racing through town.

I'm pretty sure they meant faking being a ambulance or a firefighter or a cop to rob someone.

Ah sorry, misinterpretation on my end. You are probably right.

But: fake cops robbing people is pretty common too. Unfortunately.

Even more unfortunate, real cops doing it also happens.

Yeah and I bet that it would become even more common if cars fell for it every time. A predictable victim that always for for the same trick in the same way is much more attractive than a human that may respond in many different ways, that reads the news and may become more cautious if there is a crime wave.

I think the potential issue is that the driver of a non-emergency vehicle could start gesturing in particular ways that could trigger the autonomous car to think it was an emergency vehicle.

So then you have to teach the car what all the various kinds of emergency vehicles look like, in every jurisdiction. That's hard enough, but then what happens if it's an unmarked police car, for example?

I think the point is that humans do make this judgement call. We know what a real emergency vehicle and what real emergency personnel looks like, and we know what a crazy person waving their arms looks like.

When some idiot in the opposing lanes is waving for me to take a left across multiple lanes of moving traffic, I know that I should ignore their idiotic traffic direction. I know that when there's an emergency situation with a fire fighter telling me to continue through a red light, I know that I should follow the directions.

I hope you're not suggesting like a hater that a Tesla with FSD (available next week, I promise) isn't capable of performing a snap psych eval of any random person standing at the side of the road.

I am suggesting that a car which follows all human gestures is not better than a car that follows no human gestures.

To spell it out, if an SDC Taxi stops for anyone waving frantically, how do you prevent robbers and rapists from routinely harvesting their passengers?

Exactly how you stop people from flagging a human driven taxi with plights for help and then robbing or raping the passenger. This doesn't seem like a problem specific to self driving.

Relevant xkcd: https://xkcd.com/1958/

A Taxi has a sober human driver with professional experience to handle such situations.

A piece of software with some cameras and actuators is really not the same.

self driving car AI challenge, run into the cross walk and start waving every car through

I agree with treat it as real. Then, if the car has an adult passenger you can ask them for their judgement (and responsibility for the outcome!)

If the car has no passenger. Then stop and move slowly as required/indicated. Sure, it may be a car thief, but without a passenger, it is simply another car theft.

> When you see flashing lights on the highway coming up behind you, do you think “I bet that ambulance is faking it so they can get home faster”, or do you just move over?

“You” in the specific or general sense? Because, my observations of highway behavior suggest that actual humans are…far from ideal at this.

No need for human gesture, just put a stop sign or a road closed sign. These were likely taught to the model as its ABCs

As a general principle, one can imagine special signals relevant to autonomous vehicles. But there's probably something of a chicken and egg sort of thing and I have problems with a lot of processes being required to deal with early adopters of this sort of technology.

It kinda sucks, though, to put the responsibility for this on the first responders, who already have enough to do dealing with the emergency.

There are going to be times when someone is legitimately responding to an emergency and doesn't have a stop sign handy.

And there are going to be a lot of pranksters who do, and will use them to mess with so-called self-driving cars.

One thing that is often overlooked is that software can only understand what it was trained to understand.

There's a connected vehicle protocol (DSRC) which broadcasts all kinds of metadata about your vehicle. There was stuff like "transaxle temperature" and "steering angle" in there, if I recall, so I'd be surprised if there wasn't also "emergency flashers engaged".

Perhaps autonomous vehicles should be be sensitive to that field, signaling for manual control when they're too near a vehicle which has it enabled. That way any vehicle--not just the fire department--can create a "human control only" field by activating their emergency flashers. Perhaps emergency vehicles could be recognized such that they create a much larger field.

> But how do we distinguish real emergency scenario from fake one?

You don’t. Bad actors can already pretend to be fireman doing fireman things and direct traffic. Human drivers will also be mislead by that.

What is more, if you have a high viz vest, a hard hat, and more confidence than a garden slug you can also direct trafic. Not legally mind you, but more often than not people will follow your directions. If there is a whole gaggle of fake construction people and they have cones too they can probably re-route a highway.

Whatever is stopping people from causing misschief (too often) by these “powers” doesn’t seem to be different in the case of self driving cars than with human driven cars. In fact the self-driving car is more likely to retain a perfect photographic evidence of the mischief maker’s face than a regular human driver.

How do people distinguish that? A person is likely to fall for a fake police officer and fake police car

Have you seen any of the adversarial neural net stuff? I wouldn't be surprised if it was possible to trick a car into thinking someone is a police officer without formally impersonating a cop

> Sounds like having support for interpreting human gestures will solve some of the problems.

Add that to the _LONG_ list of issues that will have to be solved until we can even think about releasing autonomous cars in the streets.

>> but I wonder how long or what changes will have to be done in current road signs, critical city infra to support them.

Unfortunately that attitude is common is the industry. The notion that the rest of the world needs to provide extra, up to date information for the shitty AI so it can function.

I have news for ya. Emergency road repairs, fires, etc are not going to make it into your world database in a timely manner so a dumb vehicle can navigate the situation. It will all be over before anyone even thinks about fixing your system for you.

Was in backroads and ran into a cop directing travel. Apparently a small fair nearby.

Dude was totally pissed and gesturing every which way wildly. When I couldn’t immediately decode. He just started swinging even more wildly. Turning redder and redder.

Was fairly sure I was going to get shot arrested. So I just did a U turn. As I was leaving the next guy started yelling at cop that he was blocking all possible routes to his town.

Teleoperations is a thing.

Simple, just transfer control over to a remote operator. We already do this for delivery rovers (see e.g. LA's coco rovers). Not an impossible problem unless all communication infrastructure is down (solar storm, hurricane, that sort of thing).

Reply on news.ycombinator.com