Horse racing is on a relatively safe enclosed track. Pigeon races are unconstrained by geography and have predators such as hawks. If horse races went through a forest or other rough terrain with significant predators such as bears it would be a better comparison.

I am not advocating for this in any way (the horses have a bad enough time with broken legs and things on traditional racing courses). Just that the two things are not like-for-like, similar to Formula 1 vs cross-country rally racing for cars.