We don't know how much of this stuff exists in the interstellar space. It's possible that a probability to hit something there is much less than ISS hitting something big enough to destroy it, even with all radars and maneuvers. So I wouldn't consider it as an argument against interstellar travel, not yet.