You're taking a decision which results in the death of those people. The trolley problem is about showing how people's attitudes change when they become responsible, instead of being bystanders.
You are seated at a console. In front of you is a button. If you don't push the button, five randomly selected people will die. If you do push the button, only one randomly selected person will die. That is all you know. Do you push the button?
Or if that still feels like a moral dilemma to you, try this:
You are seated at a console. In front of you is a button. If you push the button you will save the lives of four randomly selected people who would otherwise die. Do you push the button?
Instead, of "you must flip the switch" you get a speeding train. Instead of the switch you get fat people and bridges. Are the five people children and does the fat person have heart trouble? Is the fat person fat because he's greedy, or does he have a glandular problem. Will anyone see me pushing the fat man? And again, what guarantee do I have that a fat man's body will stop a train? How old is the fat man? How young are the kids? Are there suspicious looking people around who, if the train fails to run over the five people, will just back the train up and try again?
But you're right, four versus none is not the same as five versus one. And five versus one in the abstract is not the same as these five versus that one. The details matter. That is the point I was trying to make.