As you well know, that example is ridiculous because of the obvious magnitude difference between the two. Moral questions aren't interesting at that scale: nearly every moral system will give you the same answer.

A more interesting question from the utilitarian perspective is this: Imagine you can press a button A. If you press A 10000 people are spared the severest migraines, but one person dies. Do you press it? If yes, two people? If yes, what N is too high? Ah, easy, just sum up the total negative utility of 10000 migranes, and then the negative utility of being dead multiplied by N, and you have your answer. Laughable, of course.

The idea that people opposed to utilitarianism are shying away from their own moral responsibility is ridiculous. Utilitarianism can be used to justify obviously wrong acts, such as, again, the liquidation of small and annoying minorities in order to increase the total utility. The fact that most utilitarians would be horrified by this proposition isn't an argument in favor of utilitarianism. Rather, it is a testament to the moral common sense of even utilitarians.