You aren't hunting the baby for sport. Sport is not among your reasons for hunting the baby.
This actually devolves into human neuroscience, the more I think about it. "I want to throw a ball fast, because I want to win the baseball game". The predictive processing theory view on the statement says that the set point at the lower level (your arm) and the set point at the higher level (win the baseball game) are coherent, and desire at each level doesn't directly affect the other. Of course, you'd have to abandon a homunculus model of the mind and strongly reject Korsgaard, but that's on shaky ground scientifically anyways so this is a safe bet. You can just say that you are optimizing for your village as a higher level set point, but are hunting for game at a slightly lower level set point.
Note that sport is not a terminal desire, as well. Is a NBA player who plays for a trophy not playing a sport? Or a kid forced to play youth soccer? So you can't even just say "sport must be an end goal".
So, in your scenario – the person's initial reason for harming babies isn't their own personal enjoyment, it is because they've been coerced into doing so by an evil dictator, because they view the harm to one baby as a lesser evil than the death of their whole village, etc. And even if the act of harming babies corrupts them to the point they start to enjoy it, that enjoyment is at best a secondary reason, not their primary reason. So what they are doing isn't contravening my principle.
Anyways, I actually think your statement is incoherent as stated, if we presume moral naturalism. There's clearly different levels set points for "you", so "sole reason" is actually neurologically inconsistent as a statement. It's impossible for "sole reason" to exist. This radically alters your framework for self, but eh it's not impossible to modernize these structural frameworks anyways. Steelmanning your argument: if you try to argue set point hierarchy, then we're back to the NBA player playing for a championship example. He's still playing even if he's not playing for fun. Similarly, hunting a baby for pleasure can still be hunting for a village, as The Most Dangerous Game shows.
More generally (and less shitposty), the refined principle is now quite narrow and unfalsifiable in practice, as a no true scotsman. How would you ever demonstrate someone's "sole or primary" reason? It's doing a lot of work to immunize the principle from counterexamples.