Adversarial image attacks are no joke
unite.ai
unite.ai
It would be fairly easy to add poison to a water supply or the air intake of a large building and kill a large number of people. This rarely happens though.
It's ok that water sources, buildings, and people aren't completely immune to poison. The safety requirement isn't that poison can't hurt. Instead, we rely on weaker protections. We try to make known poisons hard to make, we try to track people who could make them, and we try to make it hard to deliver poison.
I believe the same will be true of adversarial examples for vision (and language) models. We can try to make them hard to make, hard to posses anonymously, and hard to deliver. I think this will be much easier with computer vision than with poison, so I'm not worried about it.
For example, consider the case of pasting a sticker on a speed limit sign that causes Teslas to swerve off the road. Governments should protect people from this in multiple ways, similarly to how they protect us from poison:
1. People who post these stickers should go to prison.
2. People who create and distribute these stickers knowing their purpose should go to prison.
3. Tesla should be civilly liable for cases where preventing such an incident was possible with known technology.
4. Roads should be modified over time to make it more difficult to do this attack.
It gives the illusion of security, but they would absolutely not deter a determined threat actor.
The only reason that the water supply isn't poisoned is it's unpractical for a single person to conduct the whole exploit chain: Construct the poison in enough quantities, gain access to facilities supplying the water, and actually throwing the compound in it. It's unpractical even for "underground" types. Especially the quantities required.
Mathematics and computer science is a different story in my opinion. You cannot restrict science or thought. You can try, but good luck. The most you can do is delay it. If there is an attack that enables someone to flip a Tesla on the road (as suggested below), the security theater will hide the attack from common folk, but determined actors will reach it eventually, and at that point, they can deploy it as they wish. And in contrast to the water plant, the logistical endeavor to exploit it is absolutely easy in comparison: slap a sticker on your vehicle.
Security by obscurity or by theater is rarely a good strategy in my opinion. We should absolutely be transparent about these kind of things, and allow researchers full access to develop attacks against these systems, and effectively communicate when they are found.
> In 1998, the defense filed a post-conviction motion for a new trial. At a hearing on the motion, several witnesses testified that they had driven by the intersection days before the accident and the stop sign was already down. Some of the witnesses said that after the charges were filed, they reported to both Hillsborough County Sheriff’s detectives and the prosecution that the sign had been down for days, but the information was disregarded. One of the witnesses said she spoke to the prosecutor who disregarded the report and replied that she intended to “burn their ass,” referring to the defendants.
>The motion for a new trial was denied, and the defendants appealed the decision. In March 2001, the Florida Court of Appeals reversed the manslaughter convictions of all three defendants. The court held that the prosecution had made improper comments during closing argument. The court did not reverse the grand theft convictions.
I don't think putting people to prison for, say, flipping a Tesla by screwing with its computer vision algorithm is security theatre. Rather, it's accountability. I'm pretty sure most people are aware that you cannot stop a determined attacker from breaking a system (which is exactly why Spectre mitigations were implemented as soon as the vulnerability was discovered: it's hard to exploit, but still possible).
Defining a legal code for exploiting computer systems through their hardware or their software is not security theatre, it's to ensure that we have a system to punish crime.
5 years ago it would have been pretty much unthinkable that a ransomware attack could actually take down most of the eastern US petrol pipeline infrastructure but here we are, no one prosecuted, and apparently the only thing stopping other high profile attacks is the forebearance and self-policing of the thieves themselves.
Laws against murder don't prevent murder from ever happening, but they ensure that committing it is weighed against very high costs.
Perhaps there are other ways to reduce the chance of bad things happening, like reducing opportunities for the bad thing to happen in the first place (eg. not overly relying on computer vision).
Sure. And the threat of jail/imprisonment doesn't deter determined murderer's. It doesn't mean we shouldn't put deterrents.
GP doesn't say we shouldn't, but rather that it's not good enough.
> The weak association between higher incarceration rates and lower crime rates applies almost entirely to property crime.16 Research consistently shows that higher incarceration rates are not associated with lower violent crime rates.
It does make sense. If a person is committing a crime in the hopes of material gain, reducing that material gain by imposing a negative gain if they get caught should deter them.
It doesn't seem like a person committing a crime of passion would be using that sort calculus. And it turns out in this case intuition is right: the figures say they don't. Ergo the threat of jail has no effect on the number of murders committed.
What deterrents is one of the hardest problems society has ever grappled with. How do we stop antisocial behaviours? Prisons (a modern punishment) do not seem to work, for a multitude of complicated reasons. This is coming from someone who has been through the system.
> The only reason that the water supply isn't poisoned is it's unpractical for a single person to conduct the whole exploit chain
It's a quantitative question, just like with computer vision. If you don't like the poison example, consider viral DNA, which is also dangerous in the right hands and does not require massive supply chain control. Not everyone has access to a driving dataset like Teslas, and it would be difficult to trick a Tesla without such a dataset.
We should allow researches to develop attacks, just like we should allow researchers to study poisons, DNA, and viruses.
Gaining access is rather easy. You can easily fly drones over most of reservoirs and dump whatever you want into them. Making strong poisons is also relatively easy, eg. dimethylmercury can be easily synthesized by any chemistry graduate.
If your vision system can be caused to swerve off a road by a sticker then maybe it shouldn't be used?
Human driving is full of redundancies, and there is a clear hierarchy of information. People will not rush into a road full of cars going on the other way, it doesn't matter what the signs say.
If your automated driving system doesn't have those same features, it's not ready for use.
And people would not drive into a river passing through multiple barriers, just because their GPS says so.
https://theweek.com/articles/464674/8-drivers-who-blindly-fo...
https://indianexpress.com/article/trending/bizarre/driver-in...
This common sense of danger is what IAs must have before we can trust them on the streets.
For instance in the Uber fatal crash: the IA correctly labelled the cyclist as a cyclist several times then revised its judgment, then definetely labelled it as a cyclist but “oops too late”. A human would be like “wait, I think I saw a cyclist, slow down and figure out what this really was”. Obviously with the current computer vision technology you cannot do that, otherwise your car would be afraid of everything and refuse to even start, because there's way too much mislabelling happening at some point or another.
This explains why there are only 5.25 million car accidents a year in the US.
I don't know how many automated vehicle Uber has, but with one killed already the ratio is a few orders of magnitude worse for their automated vehicle than for the average driver in the US. (probably even way worse than the average drunk driver actually)
You might want to watch the one-way roads in big cities. It happens a lot more often than you assume.
It also is (usually) self-correcting: oncoming traffic will honk, stop, or move around. The offender will (usually) realize their mistake and try to correct.
Sometimes, though, that's not enough. Searching "killed in wrong way one way" on DDG (or assumably Google) yields many (!) news stories.
So, yeah. People react. Which brings up the question: How well do self-driving AIs respond to a wrong-way driver? How well do self-driving AIs recover when they are the wrong-way driver, and they suddenly have enough data to realize that?
It's so frequent in fact that the barflies at one local place (that has a beer garden from which you can see a one way road) turned it into a drinking game - wrong way car == take a shot.
Humans are surprisingly good at driving under normal conditions.
In my experience, I've not been in an accident with a teen, nor someone elderly, though I know people that have (both causing and being involved). Neither have I been in an accident with someone that I could tell was impaired by drugs or alcohol. I don't know for sure any of them involved a phone for that matter. Weather was only a factor in one accident (pouring rain, low visibility).
I have nothing to suggest that any of my accidents were caused by anything other than inattentiveness, even the one time weather played a minor role. I also see a lot of dangerous behavior every time I drive: people running lights and stop signs, completely ignoring yield signs (seriously, they must be invisible to everyone else), failing to yield right of way, failing to signal turns and lane changes (my favorite is turning the signal on after moving into the turn lane), lots of phone usage (for everything except making a call, from maps to texting to watching videos!).
Do you just drive a lot or do you brake too late / too hard? Because an accident rate that high is rather unusual.
As I stated in my post, most of them are due to people being inattentive and not following the basics like keeping a reasonable distance checking blind spots. Others I suspect phone use, but can't prove it.
You mean live in a place where drivers tailgate?
That's way too many to take your word for it. Where there's smoke, there's fire. Well, maybe not... but you sure as shit should suspect a fire.
I wonder how well self driving algorithms would compare if tested in such a "hostile" environment? Perhaps we shouldn't allow them on more "friendly" roads until they can consistently surpass human performance under such adverse conditions.
Humans have well-developed models of how things should be, can detect when things seem wrong, and come up with ways to address the apparent anomaly (including taking steps to investigate and evaluate the situation.)
Humans do not always use these capabilities well, but they have them, while similar capabilities are at best rudimentary and fragile in current AI. The premise of this article is that these capabilities will not come easy.
But... around here at least, signs have stickers or graffiti on them often enough. Like adding the name of a politician under a stop sign: "Stop [Harper]". An appropriately made adversarial example won't stick out visually the same way that a wholesale sign swap will.
An orange diamond "detour" sign isn't easily confused for a smaller rectangle "one way" sign.
Additionally, there should always be two large "do not enter" plus two large red "wrong way" signs that are visible to a driver from in the intersection before turning.
Something as simple as tape or other coverings on an existing sign should never result in any confusion as to right-of-way for a driver paying attention.
If I write crappy paint program and all I can claim is, "It's no worse than the time/effort of drawing by hand," what exactly have I achieved in your opinion?
And if the posts on HN wrt blockchain and ML constantly feature these "no-worse-than-what-we-are-replacing" arguments while posts about, say, paint programs don't, what does that say about the buzz around blockchain and ML?
Edit: clarification
Perhaps you need to face the fact that if the CV algorithm fails against these examples when humans don't, then the CV algorithm is too brittle and should not be used in the real world. I don't trust my life to your "It kinda looks like a road, oh wait it's a pylon, I've been tricked, BAM!" dumpster fire of an algorithm.
We used to have to craft robustness into algorithms based on the false positive rate. Nobody looks at a CFAR style approach anymore, and it shows. The state of the art approach of pinning everything on ML is a dead-end for CV.
Humans are absolutely susceptible to a wide variety of adversarial attacks.
This is the tricky bit.
Night-time driving, bad weather, icy roads, bumper-to-bumper traffic: these are all situations in which some algorithms can outdo humans in terms of safety. Faster reactions, better vision (beyond what human eyes can see), and unlimited 'mental stamina' can make a big difference in safe driving.
But then there will be the occasional situation in which the CV screws up, and there's an accident. Some of those are ones where many/most humans could have handled the situation better and avoided the accident.
So how do we decide when the automated car is 'good enough'? Do we have to reach a point where in no situation could any human have done better? Must it be absolutely better than all humans, all the time? Because we may never reach that point.
And all the while, we could be avoiding a lot more accidents (and deaths) from situations the AI could have handled.
To be clear we are talking about CV which relies on passive optical sensing in the visual spectrum through cameras, not radar or lidar or IR or multi-spectral sensors.
Within this context, your statement is incorrect. A typical camera’s dynamic range is orders of magnitude lower than the human visual dynamic range. Ergo a camera sees a lot less at night compared to a human and what it does see is a lot more noisy. Note that this is the input to the detection, tracking and classification stages, the ouput of which feeds into the control loop(s). It doesn’t matter how good the control system is, it cannot avoid what the vision system cannot see.
Well, I think you mean working off RGB data? That's not necessarily the problem you have to solve even if your parts are regular cameras, as long as they're dedicated to your uses. You can modify them to see IR or polarization.
This is actually a really interesting point. I don't think people appreciate how far accident rates have actually dropped for modern cars without self driving. Even at million-cars-per-year sales rate you will need years of data to prove that a single self-driving software+hardware combo is better than humans with high statistical confidence. Your development cycles would be decades-long, like in aviation, if you want to be sure you're actually improving.
Also, you can make reasonable inferences about fatal accidents using non-fatal accidents. All fatal accidents are also normal accidents. If waymo has far fewer non fatal accidents, you can reasonably infer it would have fewer fatal accidents. Otherwise you'd have to believe waymo's accidents are more likely to be fatal, but the opposite is probably true because of the locations and speeds where they drive (at least for the passenger lol)
You can also make inferences about accidents based on disengagements or undesirable events (human labeled). It's not as data limited as you might think.
First, I assume it's already illegal to be "adversarial" to drivers. A bright light or changing signs etc already do that now. For example look at all the laser pointer stuff with planes.
Second, I don't think self driving cars are just using the softmax output of an object detector as a direct input to car control decisions. In the absence of a stop sign, the expected behavior would be common sense and caution, the same as if someone removed the sign. If the SDC logic is not robust in this way, it's not safe for many other reasons.
With this in mind, I think the situation is probably already reasonable well covered in existing regulations.
>We can try to make them hard to make, hard to posses anonymously, and hard to deliver.
To stretch your own analogy, I have a wide selection of poisons at home. Except we call them cleaning products, insecticide and automobile fluids.
You can get public support against adversarial attacks on self-driving. Except the main use case for computer vision is passive surveillance. Good luck on that front.
Oh, and just for funzies, I'll point out the irony that some of the people building CV surveillance systems would post on HN that regardless of regulation it'll exist no matter what the government wants. The argument was that it'd be so hard for the government to control CV surveillance, that law wouldn't prevent business from creating and using it anyway. When it comes to adversarial attacks, it seems more likely to involve actions of private individuals rather than businesses, and businesses minimize legal risk in a way individual citizens don't.
At a minimum, you can't modify street signs. Eg in Washington State:
RCW 47.36.130
Meddling with signs prohibited.
No person shall without lawful authority attempt to or in fact _alter_, deface, injure, knock down, or remove any official traffic control signal, _traffic device_ or railroad sign or signal, or any inscription, shield, or insignia thereon, or any other part thereof.
RCW 46.61.075.1
Display of unauthorized signs, signals, or markings.
No person shall place, maintain or display upon or in view of any highway any unauthorized sign, signal, _marking or device_ which purports to be or is an imitation of or resembles an official traffic-control device or railroad sign or signal, or _which attempts to direct the movement of traffic_, or which hides from view or interferes with the effectiveness of an official traffic-control device or any railroad sign or signal.
> researchers shouldn't be distributing them with the intent to cause harm
It could be used to cause harm and publishing your work is distributing it (by definition). Similar laws have been (and are) used to target people for unjust reasons.
Defacing street signs is already illegal.
Making various abstract drawings illegal to wear on your clothing (or print on a sticker) is a terrible idea. If the algorithm used by a product can't handle a pedestrian wearing a sweatshirt with an adversarial example on it then that product simply isn't suitable for public use.
Look at drug research. There is plenty of red tape that hinders it. Although, here, the "harm to society" is defined by the nation state.
However, I agree with your proposals in the top-level comment.
I used to think this until someone walked me through the logistics of both and made me realize that you would need an agency-alerting level of poison for the water supply and some way to avoid people just shutting off the A/C and sticking their heads out of windows (also a huge amount of gas). Also the news can't exist to alert anyone immediately.
Doesn't this fall under: "We try to make known poisons hard to make, we try to track people who could make them, and we try to make it hard to deliver poison"?
And the rest of it being you should use something odorless / tasteless.
Erm. We can maybe do something about delivery, but stopping people from making (and thus, possessing) them is virtually impossible, since all you need is an undergrad-level understanding of ML (if that) and some freely-available software.
> I worry about self-driving car safety features.
> What's to stop someone from painting fake lines on the road, or dropping a cutout of a pedestrian onto a highway, to make cars swerve and crash?
> Except... those things would also work on human drivers. What's stopping people now?
> Yeah, causing car crashes isn't hard.
> I guess it's just that most people aren't murderers?
> Oh, right, I always forget.
> An underappreciated component of our road safety system.
For instance, taking out the United States internet would probably only required 3-4 strategic bombings. I bring this up because Tennessee had one of those bombed Christmas last year -- https://www.theverge.com/2020/12/28/22202822/att-outage-nash...
> This brought down wireless and wired networks across parts of Tennessee, Kentucky, and Alabama
Most people aren't all that concerned about doing damage. Keep people happy and generally you don't have crime.
> 2. People who create and distribute these stickers knowing their purpose should go to prison.
... and you're asking the entire world to change to make your vehicle work:
> 4. Roads should be modified over time to make it more difficult to do this attack.
10 years ago when AI cars were getting memed into existence I would get dogpiled on for naysaying. Now you same people want laws to stop people from breaking what was already easily breakable and which you argued was unbreakable.
Also, your poison analogy is invalid. Poison is not uncommon because of law, it's uncommon because it's uncommon. In the future, crazy people will be poisoning random stuff in the grocer because they don't like the demographic that shops there.
Poison is not uncommon. Without 10 feet I have enough poison to kill dozens of people. Most people do.
If your garbage AI powered car can be tricked easier than an 8 year old, who's fault is it?
We wouldn't let an 8 year old drive, but your garbage AI is fine?
In most cases, our first and last defense against an attack of this type is to rely on the fact that nobody is interested in doing it.
3 doesn't seem crazy, but it would practically end up with caps, which might not be what you're looking for
4 This both: seems possible, and will basically never happen due to cost in every little jurisdiction
See Metromedia, Inc. v. San Diego for example.
#2 is expensive and difficult, but that's what we do for explosives, poisons, drugs, etc.
If I wanted to print an image and put it on a t-shirt that would trick a computer driven car into doing something if its cameras saw my shirt, that’s not my problem. The barrier to entry is much lower too so I think it’s up to the engineers to solve it instead of trying to dump the hard problems on society.
You are getting close to something with you second statement. There are laws that criminalize actions like yelling 'Fire' inside a movie theater or provoking a fight (fighting words). Essentially these laws isolate the protected 'speech' from a non-speech and therefore non-protected 'action'.
However, it would be an extreme stretch to apply or expand these to apply to simply wearing a t-shirt. There is already plenty of case law that says wearing/displaying symbols or profanity is not enough to be considered fighting words/act. Heck, in most cases just using a racial epithet is not enough to be considered fighting words and/or hate speech. [1]
At most you will ever be able to convict is if someone is installing these adversarial images on public property (e.g street signs). In that case you might be able to use the harmful nature/intent of the images to elevate what would otherwise be a vandalism charge to assault. Essentially there needs to be a distinct and meaningful 'action' beyond just wearing/expressing speech.
[1] https://www.msn.com/en-us/news/us/federal-court-saying-the-n...
Then let me change my example to show legal items being used with the intent to cause harm is still illegal. I'm free to put razors into candy, but if I hand it out on Halloween it'd be illegal.
>However, it would be an extreme stretch to apply or expand these to apply to simply wearing a t-shirt. There is already plenty of case law that says wearing/displaying symbols or profanity is not enough to be considered fighting words/act.
This hypothetical T-shirt isn't comparable to fighting words, wearing it would unquestionably cause harm to the relevant ones who encounter it. Owning or creating it might not be a crime, but wearing it in public is endangering the public.
Only because you're driving a car that was programmed by monkeys and sold by PT Barnum.
If your car can't tell the difference between a street sign and a T-shirt, it's really not fully self driving, is it?
I’d even argue that to be used on public road that any self driving code needs to be open source.
It is data you are intentionally using to hurt someone. If a person legally gained access to a water treatment plant and sent inputs to poison the water, you wouldn't say that person only sent data and the code was at fault. The shirt is inputting data into the car the same way.
Tools should be made with safety in mind, but when people use them to cause harm the person is at fault too.
Or have I been negligent by creating a device that is unable to process certain words without exploding? ie it can't do it's job properly and safely.
This hypothetical car is too badly designed to ignore things that aren't road signs.
A 5 year old can recognize if something is a road sign or not, and we don't let them drive.
That's how pathetic this car is.
Of course it shouldn't be allowed on a public road, and it shouldn't be allowed to be sold because it is unsafe.
I agree with this, that's separate from what I'm discussing.
If you made a computer that exploded when someone typed a word into it, and then I knew about this and told someone else to type that word in, we would both be culpable.
Perhaps as some technicality but in reality who is (much) more culpable? Who would think a computer would be made so poorly that it fails dangerously if the wrong word is typed in?
One could say they didn't believe it would be true because it sounds too implausible.
Same with a car that drives off the road if it sees the wrong thing on a t-shirt.
I don't think it's fine for a company to just make products that are super dangerous if some slight edge case is met. In my opinion they would be liable.
Otherwise terrorism can be legal if you slap an "AI" label on it.
The person who both knew that the computer was dangerous and directly caused someone to perform that dangerous act. Without them, it's possible no one gets injured.
>I don't think it's fine for a company to just make products that are super dangerous if some slight edge case is met.
As I keep telling you, neither do I.
I really don't understand your point of view. Of course companies shouldn't make unsafe products, but just because an unsafe product exists doesn't justify someone using that product to hurt someone. Just because the danger sounds implausible doesn't forgive the situation. It sounds implausible that touching a bit of metal would kill someone but if I made you touch a live wire I'd be at fault.
I don't think if people want to wear any shirt, even one specifically designed to fool garbage AI, that it is their fault that a car suddenly decides to kill people. If you want to sell a car it must be smart and safe enough to ignore any T-Shirt, poster or painting in it's vicinity, just like my old, non-garbage non-AI car.
If you can't do that, then you can't sell cars without being sued into oblivion. And that's the way it should be. Messing with street signs is already illegal. But wearing a shirt, no matter what it has on it, is not.
It would be an extremely difficult to enforce though.
Doing things with intent to harm others is illegal, even if you use a sticker to do it.
> Tesla should be civilly liable for cases where preventing such an incident was possible with known technology.
This is currently likely the case, but is not proven until a lawsuit happens.
This would surely not pass constitutional muster.
> Federal law prohibits the possession with intent to sell or distribute obscenity, to send, ship, or receive obscenity, to import obscenity, and to transport obscenity across state borders for purposes of distribution.
A specific recent case:
https://www.mtsu.edu/first-amendment/article/167/united-stat...
> “offers to engage in illegal transactions are categorically excluded from First Amendment protection.”
> If it's illegal to posses something, the government can ban offering to sell and distribute it.
This is begging the question, because first you would have to show that banning the mere possession of adversarial images doesn't violate the First Amendment. Otherwise you can make it illegal to possess books in a way to prevent the sale and distribution of literature.
It seems to vary. Typically not, and when attempts are made court verdicts seem to be mixed. But unfortunately sometimes people do get convicted and the convictions upheld. For example, United States v. Whorley. https://caselaw.findlaw.com/us-4th-circuit/1431669.html
> offers to engage in illegal transactions
Just to clarify, that means "offers to engage in crime". Other than the aforementioned obscenity laws freedom of expression is generally quite well protected in the US so there won't be a crime in the first place (and thus related transactions won't be illegal).
> If it's illegal to posses something, the government can ban offering to sell and distribute it.
I think you misunderstand slightly. If it's illegal to possess something then it is _already_ (to the best of my knowledge) illegal to offer to sell or distribute it. The question is what the government is and isn't allowed to ban possession of.
Interestingly, in the case of "obscene" materials possession itself isn't banned. Only import, sale, and distribution.
The issue with "People who create and distribute these stickers knowing their purpose should go to prison." is that such a wording seemingly bans them outright regardless of intent. That is an affront to freedom of expression. If I want to craft adversarial examples I shouldn't need to justify my intentions and seek permission up front. An action should need to be justified as illegal on a case by case basis, not the other way around.
Note that it is already illegal to deface street signs so by extension if you create and distribute stickers with the express intention that they be used that way then presumably you are already violating the law today. On the other hand, such stickers are not (currently or ever, I hope) inherently illegal in and of themselves.
It's not obviously true that the illegality of possession automatically implies the illegality of the offer to sell. Consider for example a case where the seller doesn't actually possess the item, but merely offers to sell it. In this case, the supreme court upheld a law making such offers illegal. Without the ruling, you might argue that the mere offer is protected speech.
The point of all this is really indirect though. I'm just saying that in cases where "speech" concerns illegal items, like nuclear weapons, child pornography, and drugs, sometimes the government is given more leeway in controlling speech related to items.
And if they actually do hurt someone, I imagine they would be criminally liable.
Actually no. We know that only some psychopaths would do that and so the risk is minimal.
AI is currently simply not 'good enough' to be used in critical environments. The problem is that _any_ sticker or even dirt or snow or ... on any road sign can lead to misinterpretation, you can never proof that it's safe.
The risks surrounding vegetable packaging and distribution are well understood, readily quantifiable, and possible to mitigate.
Computer vision algorithms on the other hand are poorly understood black boxes with seemingly arbitrary failure modes. We do not (yet) appear to understand how to quantify or mitigate the associated risks. The consequences of failure are quite severe in comparison to food poisoning. Only a fool would trust their life to them.
Translation: everyone else in the universe is responsible for solving my problem, and also I am not responsible for solving my problem, but i do want to profit from the current state of everything being broken all the time, and, i tell my family to keep their hands on the wheel
Why does the existence of these attacks change the threat landscape at all? If people are already not doing "dumb" attacks like just changing/removing road signs why would they start doing them?
The risk of messing with road signs and throwing off autonomous vehicles really has less to do with adversarial image attacks and more to do with envisioning an impractically brittle system where the decision to stop is based purely on presence/absence of a stop sign and not on a system that has a more general sense of collision-avoidance and situational awareness (like humans do).|
Stepping back more generally, I have still never seen a case where the undetectability of adversarial attacks actually means there is a practical difference to security or safety. If you really think through the impact in the real world, usually the risk is already there: you can just change the input to the image and get bad results, it doesn't affect much that the image is imperceptibly changed. Because the whole point of using an automated vision system is usually that you want to avoid human eyes on the problem.
Because you have to physically do it, as opposed to hacking from anywhere else on the planet.
> not on a system that has a more general sense of collision-avoidance and situational awareness (like humans do).
Are vision systems to that point yet when it comes to driving vehicles?
> Because the whole point of using an automated vision system is usually that you want to avoid human eyes on the problem.
And the point of hacking an automated system is that it's easier to do that remotely than to cause a human to crash locally.
My impression is that the adversarial image attacks in question involve physically placing a sticker on something which will be in the view of self-driving cars -- it's not a remote exploit.
In a given span of time, how many street signs can a single person swap out versus how many stickers can they apply?
When sourcing the materials for an attack, how expensive are stickers relative to physical signs?
After one accident, the sticker will be removed.
Industrial sabotage to take out one car’s camera system? Ok - but which car company will do that? It’s mutually assured destruction if the other actors retaliate, and serious legal fees if caught.
High school pranks? Sure. But again, they will be identified, finger printed, the printed item will be analyzed and reviewed for which printer printed it, and the person will be ID’d.
That's not to say that either attack is less harmful than the other! If you train an image classifier to find bikers, it's not really wrong or right to say that a picture of a biker qualifies. But if a car stops lest it run over a painted bike on the road, that's obviously bad. The problem is that you aren't trying to recognize bikers, you're trying to avoid obstacles. We just don't train well for that.
We feed a system a series of images of bikes and then select the ones that can pick out a bike but we don't know how the bike is being chosen. We know it is picking out bikes but we have no way to predict if the system is picking out bikes or picking out a series of contrasting colour and shadow shapes and could easily be thrown off by anything that contains the same sort of data.
It’s too bad you can’t analyze brains like you can with neural networks. It’s trivial to visualize filters and feature maps or to create heatmaps showing which pixels (shadow shapes?) in a specific image affect the classification output and why (contrasting color?).
> which pixels (shadow shapes?) in a specific image affect the classification output and why (contrasting color?)
Sure, you can watch the Rube Goldberg machine work. It doesn't mean you understand why it works on a conceptual level or have any hope of rigorously quantifying when and how it could fail.
The big question that remains is - so what? There's exceedingly few use cases where the existence of adversarial examples causes a security threat. There's a lot of research value in understanding adversarial examples and what that tells us about how models learn, generalize, and retain information, but I am not convinced that these attacks pose a threat remotely close to the amount of attention given.
The classic example of a sticker on a stop sign is, in my view, more of a dramatization than a real threat surface. Designing an adversarial perturbation on a sticker that can cause misclassifications from particular angles and lighting conditions is possible, but that alone won't cause a vehicle to ignore traffic situations, pedestrians, and other contextual information.
Plus, if I wanted to trick a self driving vehicle into not stopping at an intersection, it would be much easier and cheaper for me to just take the stop sign down :)
Like billboard with stop sign on it.
I don't think that "cause an air to fail to stop" is the correct threat to address, I think "making AI stop and therefore cause traffic" is.
Wake me up when I can have any two arbitrary addresses as start and end points and a machine or computer can drive me between them, 24/7/365 - barring road closures or whatever.
Basically they need to improve their driving software some 10 000x times. From driving 100km before safety critical disengagement to 1 million kilometers. 1 - 2 million milles is benchmark presented by CJ Moore, Tesla’s director of autopilot software to California Department of Motor Vehicles.
> “Tesla is at Level 2 currently. The ratio of driver interaction would need to be in the magnitude of 1 or 2 million miles per driver interaction to move into higher levels of automation. Tesla indicated that Elon is extrapolating on the rates of improvement when speaking about L5 capabilities. Tesla couldn’t say if the rate of improvement would make it to L5 by end of calendar year.”
If they manage to keep on doubling distance driven every 6 months then we should be there in:
log2(10000) * 6 months = 8 years
You can make your own predictions here: https://www.metaculus.com/questions/5304/widely-available-te...
This isn’t an eventuality, it’s the current state of the industry.
This right here is the real underlying long term danger:
> the most popular CV datasets are so embedded in development cycles around the world as to resemble software more than data; software that often hasn’t been notably updated in years
I think a study of the failure modes of CNNs shouldn't be interpreted as an all-or-nothing evaluation of the technology as a whole, but rather a step towards gaining some confidence regarding its reliability. A lot more work needs to be done before I will trust it to drive my car.
Regarding the use of CNN's for autonomous driving, I think it is insane that people are trying to do this by trying to solve a VERY hard problem, i.e. making a machine that can do what the human brain does. Your neural net does not have enough labels to account for all possible scenarios. Instead, it would make more sense to redesign the infrastructure in a way that bounds the problem space. The current system is designed for human drivers. We should make a system that is easy to interpret for both human and machine drivers. Of course this infrastructure would benefit all car makers, not just the first mover.
That's why the systems are more robust than you probably think to failures in perception. It's also why these systems sometimes fail in ways that humans would never fail.
To your second point, I think we might agree that in order to be more robust to failures in perception, it would be good to understand where the failure modes live. I personally think we need a better understanding than we have today.
Lawyers.
If you are selling a product or service that has been trained on a dataset that contains copyrighted photos you don't have permission to use and I can "prove it" enough to get you into court and into the discovery phase, you are screwed. I'll get an injunction that shuts you down while we talk about how much money you have to pay me. And lol, if any of those photos of faces was taken in Illinois, we're going to get the class-action lawyers involved, or bury you with a ton of individual suits from thousands of people.
That link at the bottom about a "safe harbor" you get from using old datasets from the Wild West is not going to fly when you start selling.
But if the AI model just spits out copyrighted material verbatim then that is still owned by the actual copyright holder.
It references the fair use doctrine in a way that is not fully analogous to this type of use and mentions the Google books case. It also mentions that this is not settled law. It's clear that the author wants it to be fair use, but that might cloud their analysis.
Keep in mind that Google was scanning books that the legitimate owner of the physical books gave them permission to scan. If I buy a book and want to use it to train my model, fair use says I am free to do so. If I grab an unauthorized torrent of a training set, itself containing images were not legitimately purchased or licensed, there is absolutely no case law that I know of that says it is ok. I have to spend my money on lawyers trying to argue that I'm in the clear with no guarantee of success.
Maybe I'm wrong - I'd love to hear a convincing argument to the contrary!
If you are using some torrent of a dataset, nobody is indemnifying you, and once you get to the discovery phase of a lawsuit, they are going to know that you intentionally grabbed a dataset you knew you shouldn't have had access to. Treble damages!
>The second-most frequent complaint is that the adversarial image attack is ‘white box’, meaning that you would need direct access to the training environment or data.
The training data will be leaked. Companies are very bad at classifying what is and isn't private information that they need to keep secret. But anyway you probably don't even need the training data.
[3] is another paper I recommend for anyone wanting to USE CNNs for applications and wants to calmly assess the risk associated with adversarial examples
Now, from a research perspective they are fascinating, they highlight weaknesses in our ability to train models,are a valuable tool to train robust CV models in the low data regime and have paved the way towards understanding the types of features learned in CNNs (our neighbours just released this [4] which in my eyes debunked a previously held assumptions that CNNs have a bias towards high frequency features, which is a fascinating result).
But for anyone wanting to use the models, you shouldn't worry about them because you shouldn't be using the models for anything critical in a place where an attack can happen anyway. The same way that "what is the best way to encrypt our users passwords so they cannot be stolen" is the wrong way to approach passwords "how can we make the deep neural network in the application critical path robust against targeted attack" is (for now) the wrong way to approach CV.
[1] https://arxiv.org/abs/1802.06806
[2] https://www.forbes.com/sites/bradtempleton/2021/02/09/califo...
[3]https://arxiv.org/abs/1807.06732
[4] https://proceedings.neurips.cc/paper/2020/hash/1ea97de85eb63...