DN: "It sped up when we left the freeway instead of slowing down on the exit, that was dangerous. Why did that happen?"
ENG: "Car saw it was on a straightaway. We'll add a rule to handle that."
DN: "So you have to add rules for every possible situation? Doesn't that mean that the car is always at risk for what it doesn't know about yet?"
ENG: "That's Not-A-Problem. We will classify everything."
Among other things, though, these types of attacks make a lot of assumptions. For example, they assume the only input to "what is that" is a classifier that looks at the image.
Given simple data and previous classification of the image, for example, one can easily determine "hey does it make sense for an added lane sign to appear at a 4 way intersection with no apparent added lane".
Heck, you don't even need to go that far. Given previous, before vandalism classification of the sign, and no change in any terrain/mapping data, ...
So yes, i'd pretty much say "there is no need to worry about such physical attacks", as long as you are not directly hooking up an image classifier to the steering wheel. The likelihood that this ends up a major problem for self driving cars seems pretty low.