This argument is only correct under the assumption that you are the only person on earth making this decision. It is like arguing that you shouldn't go voting because the chance that your vote changes the result is, for all practical matters, zero.
Your analogy doesn't really work for me, because I do think the effect of voting for an individual, and therefore its moral outcome, is mostly irrelevant. I vote because I like the action in itself.
You say it sounds like a justification for acting against one's values - I'd put it in reverse: values, much like rights, are a system for deciding what to do when the uncertainty or cost of making a decision is too great. Blindly following them is folly if you actually want to avoid harming people.