> Utilitarianism assumes we have all the facts, yet we never do.
All the facts, while helpful, are not necessary for utilitarianism.
> "effective altruism"
I think you're attacking a straw man. You won't find Peter Singer attacking people for engaging in suboptimal charity.
> there's this underlying assumption/insult that if you're not utilitarian, then it means you don't quite understand that 5>1, and therefore you're beyond stupid and shouldn't be part of the conversation.
You can certainly say you'd sacrifice five to save one, but you need to back it up. The moral indulgence critique says, more or less, if you refuse to do it because you don't want to do something distasteful you're being a coward, not stupid.
Rule utilitarians can coherently refuse to pull the switch. I happen to disagree with their moral framework, but they can mount a vigorous defense of the position. Personally I come down hard on the side of rejecting the status quo bias. Commiting to symmetry in moral decision making is useful for avoiding contradictions. I also don't care much for act utilitarianism, since it's susceptible to non-utilitarians putting their fingers on the scale (ex. "buy this magazine or we'll shoot this dog".)