Disease is actually the main reason.

Any parasite, bacteria, virus, and especially prions spread much much more easily the closer the food is to your own biology.

Cannibalistic tribes have had issues with this in even the recent past.