What should matter is when humanity begins, or, sapience. There's no world in which it makes scientific sense to afford a small cluster of cells without anything even resembling a cow's brain (remember cows? the things we legally slaughter by the hundreds of thousands every day?), let alone a human brain, more rights than the woman the organs of whom are supporting it.
Not only that, but we don't allow the harvesting of organs post mortem without explicit, written consent, even if someone could use them to live. If we're not allowed to use other people's organs to ensure survival, even if the other person is dead, that should apply just as much to fetuses.
There's very obviously a correct side if you want to apply any semblance of logic, reasoning, and thought to the issue. There isn't one if you want to apply religion to it, and follow your gut to conclude that your all-powerful, all-benevolent god is putting souls in fetuses he knows will be aborted, taking them up to heaven for eternal happiness, and getting mad at us for it.