The fancy name is "false dichotomy", and you're completely right.