I'm not sure how else this would work. Do you mean the system may see that there's an obstacle, but doesn't know what it is, so it ignores it?
If that's the case, then fixing it is nearly an impossibility. Certainly you wouldn't want to slam the brakes for a plastic bag floating in the wind, but if every obstacle is treated the same, that's what you'll end up doing. But if you try to build a system that can make a decision on every kind of obstacle, it's an extremely exhaustive list.